Systems, methods, and devices for biophysical modeling and response prediction

ABSTRACT

Various systems and methods are disclosed. One or more of the methods disclosed uses machine learning algorithms to predict biophysical responses from biophysical data, such as heart rate monitor data, food logs, or glucose measurements. Biophysical responses may include behavioral responses. Additional systems and methods extract nutritional information from food items by parsing strings containing names of food items.

CROSS-REFERENCE

This application is a continuation application of InternationalApplication No. PCT/US19/63788, filed on Nov. 27, 2019, which claimspriority to U.S. Provisional Patent Application No. 62/773,117, filed onNov. 29, 2018, U.S. Provisional Patent Application No. 62/773,125, filedon Nov. 29, 2018, U.S. Provisional Patent Application No. 62/773,134,filed on Nov. 29, 2018, each of which is incorporated herein byreference in its entirety.

BACKGROUND

Many biological systems, including the human body, can function with ahigh degree of complexity. Further, while a single species can have anoverall general likeness, there can be significant variability betweenindividuals. Consequently, it can be difficult to understand, let alonepredict various biological responses on an individual basis.

Understanding human biological responses on an individual basis canprovide various health and quality-of-life benefits. Such anunderstanding can enable an individual to make better choices to improvetheir health. When such choices are made by a population, the overallhealth of society can benefit. In addition, such an understanding canempower an individual to better alter their lifestyle in the pursuit ofpersonal goals.

One human biological response of increasing interest is nutrition, andblood glucose levels resulting from eating in particular. Failure tomaintain blood glucose levels in acceptable levels over time may resultin adverse consequences, including pre-diabetes or Type 2 diabetes.However, individuals can vary in blood glucose response, diet, behavior,and numerous other factors. Accordingly, conventional models (e.g.,linear blood glucose response models) can be inadequate understanding anindividual's personalized blood glucose response.

SUMMARY

The present disclosure provides systems and methods that can acquiresensor and other data that records subject actions and that utilizereinforcement learning to predict a subject response and the modify thesubject's response to achieve a goal. The prediction can be a predictionof a biophysical response and/or behavioral response. Embodiments canutilize custom variational encoding to model subject actions andresponses. In some embodiments, the systems described herein cangenerate a recommendation for a subject based on a reward function andthe subject's historical actions.

In one aspect, the present disclosure provides a computer-implementedmethod for training and using a reinforcement learning algorithm togenerate a recommendation that aids a subject in maintaining oradjusting or optimizing a glucose level of the subject with respect to areward function. The method can comprise: until a convergence conditionis achieved, iteratively: (i) generating the recommendation using thereinforcement learning algorithm, which recommendation comprises arecommended meal or physical activity, (ii) processing therecommendation using a biophysical reaction model to generate apredicted glucose response of the subject to following therecommendation, and (iii) applying a reward function to the predictedglucose response to generate a first reward and updating thereinforcement learning algorithm based on the first reward. The methodcan further comprise providing the recommendation to the subject;measuring a glucose response of the subject to following therecommendation; and applying a second reward function to the measuredglucose response to generate a second reward and updating thereinforcement learning algorithm based on the second reward. In someembodiments, the method comprises using the glucose response of thesubject to train the biophysical reaction model. In some embodiments,the method comprises encoding the glucose response of the subject into alow-dimension latent space for providing to the biophysical reactionmodel and to the second reward function. In some embodiments, the firstreward function is the same as the second reward function. In someembodiments, the biophysical reaction model includes at least one bodymodel configured to generate a simulated biophysical response of thesubject in response to a plurality of inputs. In some embodiments,generating the predicted glucose response of the subject comprises:applying the glucose response of the subject to a predictor trained toinfer a future glucose response; applying the recommendation to anadherence model configured to evaluate how closely the subjects willfollow the recommendation; and selectively applying outputs of thepredictor and the adherence model as the plurality of inputs to the bodymodel. In some embodiments, generating the predicted glucose responsefurther comprises applying the simulated biophysical response to anautoencoder and generative adversarial network to generate the predictedglucose response. In some embodiments, the convergence condition isbased at least in part on the magnitude of the first reward.

In another aspect, the present disclosure provides a method that cancomprise: training each of a plurality of different autoencoder (AE)temporal convolutional neural networks (CNNs) on historical time seriesdata from one of a plurality of different data sources, wherein theplurality of different data sources comprises a continuous glucosemonitor, a heart rate monitor, and a source of food data; generating aplurality of seed values using the plurality of AE temporal CNNs inresponse to current data from the plurality of different data sources;configuring each of a plurality of different CNN encoders with one ofthe plurality seed values from a corresponding AE temporal CNN; applyingpast the historical times series data from the corresponding datasources to the temporal CNN encoders to generate encoded data values;and applying the encoded data values to a forecast configured togenerate predicted data values corresponding to one of the data sources.In some embodiments, training the plurality of AE temporal CNNsincludes: training a first AE temporal CNN with data from a first typesensor that reads a first biophysical response, and training a second AEtemporal CNN with data from a second type sensor that reads a secondbiophysical response; and applying the encoded data values to generatepredicted first biophysical responses.

In another aspect, the present disclosure provides a method that cancomprise: receiving reference values over time from a first biophysicalsensor that represent at least one biophysical response of a subject,the time including a first time period followed by a second time period;receiving first data values from a second biophysical sensor; inferringfirst predicted values for the at least one biophysical response with afirst subject model using at least first data from the second timeperiod; inferring second predicted values for the at least onebiophysical response with a second subject model using at least firstdata from the first time period; and comparing the first and secondpredicted values to the reference values to determine the accuracy ofthe subject models. In some embodiments, inferring second predictedvalues includes: inferring predicted first data values for the secondtime period from first data values of the first time period, andapplying the first predicted data values for the second time period tothe second subject model. In some embodiments, the method furthercomprises: receiving second data values; inferring first predictedvalues for the at least one biophysical response further includes usingsecond data from the second time period; and inferring second predictedvalues for the at least one biophysical response further includes usingsecond data from the first time period. In some embodiments, the methodfurther comprises: in response to at least the reference values andfirst data values from the first time period, adjusting parameters inthe second subject model. In some embodiments, the method furthercomprises: applying the first predicted values for the at least onebiophysical response to a first generative adversarial network togenerate first adjusted predicted values; applying the second predictedvalues for the at least one biophysical response to a second generativeadversarial network to generate second adjusted predicted values; andcomparing the first and second adjusted predicted values to thereference values to determine the accuracy of the generative adversarialnetworks. In some embodiments, the method further comprises: in responseto comparing the first and second predicted values to the referencevalues updating at least the first and second body models. In anotheraspect, the present disclosure provides a system that can comprise: afirst data prediction model configured to generate predicted first datavalues for a second time period from first data values for a first timeperiod which precedes the second time period; a second data predictionmodel configured to generate predicted second data values for the secondtime period from second data values for the first time period; a firstsubject model comprising at least in part an artificial neural network(ANN) configured to infer a first predicted biophysical response fromfirst and second data values from the second time period; a secondsubject model having a same structure as the first body model configuredto infer a second predicted biophysical response from predicted firstand second data values for the second time period; and a compare systemconfigured to compare a reference biophysical response for the secondtime period to the first and second predicted biophysical responses. Insome embodiments, the first and second data prediction models compriselong short-term memory networks. In some embodiments, the system furthercomprises: a third data prediction model configured to generate apredicted reference biophysical response for the second time period fromthe reference biophysical response for the first time period. In someembodiments, the system further comprises: a parameter estimatorconfigured to update at least the second subject model in response to areference biophysical response, and the first and second data valuesfrom the first time period. In some embodiments, the system furthercomprises: a feedback generator configured to selectively adjust any ofthe first data prediction model, a second data prediction model, thefirst subject model, the second subject model in response to comparisonresults from the compare system.

In another aspect, the present disclosure provides a method that cancomprise: during a first time interval, training a biophysical modelcomprising an artificial neural network (ANN) to generate a simulatedbiophysical response of a subject from at least first sensor data andsecond sensor data, the first sensor data comprising continuous glucosemonitoring data and the second sensor data comprising heart ratemonitoring data, which training comprises estimating personalizedtime-varying parameters of the biophysical model; during a second timeinterval, generating the simulated biophysical response of the subjectin real time using the trained biophysical model, the real-time dataincluding the second sensor data but not including the first sensordata. In some embodiments, training the biophysical model includes:applying at least the first sensor data to the biophysical model, andupdating the biophysical model using a parameter estimator thatevaluates the simulated first biophysical response with respect to asubject's actual first biophysical response. In some embodiments,training the biophysical model includes applying the first sensor dataand the second data to the first biophysical model, the second databeing different than the first biophysical response. In someembodiments, the second data comprises data logged by the subject.

In another aspect, the present disclosure provides a system that cancomprise: a first biophysical model comprising an artificial neuralnetwork (ANN) configured to derive network parameters in response totraining with data from at least one type of sensor to output asimulated first type biophysical response; a parameter estimatorconfigured to update parameters of the first biophysical model inresponse to the simulated first type biophysical response and an actualfirst type biophysical response of a subject; and a second biophysicalmodel comprising an ANN, configured with the network parameters,operable to infer predicted first type biophysical responses inreal-time for the subject in response to sensor data received from theat least one type of sensor in real-time. In some embodiments, theactual first type biophysical response is recorded with a first typesensor and the at least one type of sensor is different than the firsttype sensor. In some embodiments, the at least one type of sensorrecords a second type biophysical response different from the first typebiophysical response. In some embodiments, the actual first typebiophysical response is recorded with a first type sensor and the atleast one type of sensor is different than the first type sensor. Insome embodiments: the first biophysical model derives the networkparameters in response to training with data from at least one type ofsensor and data from a second source; and the second biophysical modelinfers predicted first type biophysical responses in response to sensordata received from the at least one type of sensor in real-time and datafrom the second source received in real-time. In some embodiments, thesecond source comprises data logged personally by the subject. Inanother aspect, the present disclosure provides a method that cancomprise: receiving time series data from a plurality of differentsources that each record data of a different type for at least onesubject that performs actions, the different sources including a glucosesensor that records a glucose response of the at least one subject;executing unsupervised learning on the time series data with at leastone encoding artificial neural network (ANN) to produce encoded valuesin a resulting latent space having a predetermined distance from oneanother; selecting orthogonal values based on the latent space; decodingthe orthogonal values with an ANN having a corresponding decodingstructure to the encoding ANN to generate decoded values; and mappingthe decoded values to subject actions. In some embodiments, executingunsupervised learning includes autoencoding the time series data. Insome embodiments, autoencoding the time series data includesautoencoding with a temporal convolutional neural network (NN)variational autoencoder. In some embodiments, the method furthercomprises filtering the decoded values based on relevance criteria for aparticular subject. In some embodiments, the different sources furtherinclude a second type sensor that records a second type biophysicalresponse of the at least one subject. In some embodiments, at least onesubject action is selected from physical activities of the subject andthe ingestion of food by the subject. In some embodiments, at least oneof the different sources is selected from the group of accelerometerdata, calendar data of the subject, and sleep state of the subject.

In another aspect, the present disclosure provides a system that cancomprise: an encoder configured to encode time series data values intoencoded values in a latent space having a predetermined metric distancefrom one another, the time series data being from a plurality ofdifferent data sources that record features of at least one subject, atleast one time series data being for a first type biophysical responseof the at least one subject; a value selector module configured todetermine orthogonal values from the encoded values; an decoder having adecoding structure corresponding to the encoder and configured togenerate decoded values from the orthogonal values; and an actionmapping module configured to map the decoded values to actions of the atleast one subject. In some embodiments, the autoencoder comprises atemporal convolutional NN variational autoencoder. In some embodiments,the system further comprises a filtering module configured toselectively discard some of the decoded values based on relevancecriteria for a particular subject. In some embodiments, at least anotherof the time series data is for a second type biophysical response of theat least one subject. In some embodiments, the data sources include acontinuous glucose meter, heart rate monitor and food data logged by theat least one subject.

In another aspect, the present disclosure provides a method that cancomprise: creating a data object in a system memory of a computingsystem; copying data into the data object; by execution of a decoratorfunction, transforming the data object into a data processing objecthaving an egress messaging function; processing the data of the dataprocessing object with one of a plurality of different machine learningprocesses; and upon completing the processing of the data, returning aprocessing result and executing the egress messaging function. In someembodiments, the plurality of different processes are asynchronousprocesses, and wherein the method further comprises, upon receiving amessage for the egress messaging function: creating a next data objectin the system memory; copying next data into data object; by executionof the decorator function, transforming the next data object into a nextdata processing object having the egress messaging function; andprocessing data of the next data processing object with one of themachine learning processes. In some embodiments, the processing resultcomprises a dictionary object that maps keys to values. In someembodiments, the data comprises input data to an artificial neuralnetwork (ANN) for learning operations on the ANN. In some embodiments,the data comprises input data to an artificial neural network (ANN) forinference operations on the ANN.

In another aspect, the present disclosure provides a system that cancomprise: a data store configured to store data for processing; systemmemory; a multiprocessing module configured to execute a plurality ofmachine learning processes in parallel; and a data object decoratingfunction comprising instructions executable by the multiprocessingmodule and configured to: create a data object in the system memory,copy data into the data object from the data store, transform the dataobject into a data processing object having an egress messagingfunction, and instantiate one of the machine learning processes toprocess the data of the data processing object and return processingresults and execute the messaging function to return a message. In someembodiments, the multiprocessing module is resident on a server. In someembodiments, the multiprocessing module is distributed over a pluralityof servers. In some embodiments, the machine learning processes includean artificial neural network (ANN). In some embodiments, the ANN isselected from the group consisting of autoencoders (AEs), generativeadversarial networks (GANs), long short-term memory networks (LSTMs),convolutional neural networks (CNNs), and reinforcement learning (RL)algorithms.

In another aspect, the present disclosure provides a method that cancomprise: creating a biophysical model with at least one machinelearning architecture to predict a first biophysical response, whereinthe biophysical model has been trained with at least primary sensor dataand secondary sensor data, the primary sensor data capturing a firstbiophysical response, the secondary sensor data capturing a secondbiophysical response; in response to at least the secondary sensor dataand not the primary sensor data, predicting a first biophysical responseof the subject with the biophysical model; determining if the predictedfirst biophysical response is outside of predetermined limits; and ifthe predicted first biophysical response is outside of predeterminedlimits, transmitting at least one recommendation to the subject, the atleast one recommendation selected to adjust the subject's actualbiophysical response to be within the predetermined limits. In someembodiments, the method further comprises setting the predeterminedlimits according to the subject's health status. In some embodiments,the method further comprises setting the predetermined limits accordingto a subject's health goals. In some embodiments, the first biophysicalresponse includes a glucose response of the subject. In someembodiments, the second biophysical response includes a heart rate ofthe subject. In some embodiments, the biophysical model is also trainedwith data logged by the subject. In some embodiments, the primary sensordata is generated from a continuous glucose monitor; the secondarysensor data is generated from a heart rate monitor; and the data loggedby the subject is food eaten by the subject. In some embodiments, the atleast one recommendation is selected from a physical activityrecommendation and a food recommendation. In some embodiments, thebiophysical model comprises an artificial neural network configured asan autoencoder. In some embodiments, the biophysical model comprises anartificial neural network configured as at least a long short-termmemory (LSTM) configured to predict a first biophysical response from arecommendation. In some embodiments, the biophysical model comprises anartificial neural network configured as at least one temporalconvolutional neural network configured to predict a first biophysicalresponse of the subject. In some embodiments, the at least onerecommendation is selected from a recommendation set including canonicalactions derived by autoencoding heterogenous sensor data. In someembodiments, the method further comprises, if the predicted firstbiophysical response is not outside of predetermined limits,transmitting a predetermined message. In some embodiments, thepredetermined message is selected from the group of: an encouragementmessage and a reward. In some embodiments, the method further comprisesdisplaying the at least one recommendation on a subject device. In someembodiments, the method further comprises capturing at least thesecondary sensor data with an application executable on a subjectdevice. In some embodiments, the method further comprises capturing theprimary data with the application.

In another aspect, the present disclosure provides a method that cancomprise: training a glucose regulation model having at least one firstparameter to predict glucose levels in response to at least food sourcedata; in response to information on a subject, substituting the at leastone first parameter with at least one personalized parameter in theglucose regulation model to create a personalized glucose regulationmodel; and applying food source data from the subject to thepersonalized glucose regulation model to predict a glucose level of thesubject. In some embodiments, the glucose regulation model includes atleast one neural network. In some embodiments, the glucose regulationmodel includes at least one statistical model selected form the groupconsisting of: a long short-term memory neural network and recurrentneural network. In some embodiments, the glucose regulation modelincludes at least one neural network trained with data of apredetermined population. In some embodiments, the at least one firstparameter comprises an insulin resistance parameter. In someembodiments, the glucose regulation model includes at least one glucosemodel selected from the group consisting of: a differential equationmodel of glucose regulation and a glucose model comprising a set ofcoupled equations. In some embodiments, the at least one differentialequation model of glucose regulation includes a food source function. Insome embodiments, the method further comprises: training the food sourcefunction with at least training data selected from the group consistingof: glycemic responses of a population to predetermined foods, andglycemic responses calculated from data for predetermined foods. In someembodiments, the method further comprises generating the personalizedparameters of the subject by recording a glucose response of the subjectwith a glucose meter. In some embodiments, the method further comprisesgenerating the personalized parameters of the subject by classifying thesubject into a demographic equivalent group based on characteristic dataof the subject.

In another aspect, the present disclosure provides a system that cancomprise: a computing system comprising a glucose prediction modelcomprising at least one model parameter operable to predict glucoselevels in response to at least food source data; a model parameter inputconfigured to receive at least one personalized parameter as the atleast one model parameter, the at least one personalized parametergenerated in response to data of a subject; and a food source data inputconfigured to apply food source data to the glucose prediction modelwith the at least one personalized parameter to predict a glucose levelof the subject. In some embodiments, the glucose regulation modelcomprises a neural network. In some embodiments, the glucose regulationmodel comprises a statistical model selected from the group consistingof: a long short-term memory neural network and recurrent neuralnetwork. In some embodiments, the glucose regulation model comprises atleast at least one neural network trained with data from a predeterminedpopulation. In some embodiments, the at least one model parameterincludes an insulin resistance parameter. In some embodiments, theglucose regulation model is derived with supervised training based on atleast one model of glucose regulation selected from the group consistingof: a differential equation model of glucose regulation and a glucosemodel comprising a set of coupled equations. In some embodiments, the atleast one differential equation model of glucose regulation includes afood source function. In some embodiments, the food source functioncomprises at least one neural network trained with training dataselected from the group of: glycemic responses of a population topredetermined foods, and glycemic responses calculated from data forpredetermined foods. In some embodiments, the system further comprisesan electronic device configured to generate the food source data. Insome embodiments, the system further comprises a memory coupled to themodel parameter input and configured to store the personalizedparameters.

In another aspect, the present disclosure provides a method that cancomprise: training, on a plurality of attributes, a first neural network(NN) to impute a first subset of the plurality of attributes from asecond subset of the plurality of attributes; training a second NN topredict a target value from the first subset of the attributes and thesecond subset of attributes; receiving a subset of input attributes of aplurality of input attributes from a subject; using the first NN toimpute remaining input attributes in the plurality of input attributes;and processing the first subset of inputs attribute and the remaininginput attributes with the second NN to predict a target value. In someembodiments, the first NN comprises an autoencoder. In some embodiments,the second NN comprises a bidirectional recurrent NN. In someembodiments, the recurrent NN is a long short-term memory NN. In someembodiments, the second subset of the plurality of attributes and thesubset of input attributes comprise nutrition data for food; and thepredicted target value is a glycemic value.

In another aspect, the present disclosure provides a system that cancomprise: a subject data input configured to receive input attributesfrom a subject; a first neural network (NN) trained to impute relatedattributes from input attributes by randomly selecting attributes fromsets of attributes having associated target values, and configured tosequentially receive the input attributes; a second NN trained topredict a target value from related attributes and the input attributes,and configured to receive the input attributes and the relatedattributes generated by the first NN; and a subject data outputconfigured to output and update a predicted target value from the secondNN in response to the application of each input attribute to the firstand second NNs. In some embodiments, the system further comprises: adata store configured to store training input attributes andcorresponding training target values for training the first and secondNNs. In some embodiments, the first NN comprises an autoencoder. In someembodiments, the second NN comprises a bidirectional recurrent NN. Insome embodiments, the recurrent NN is a long short-term memory NN.

In another aspect, the present disclosure provides a method that cancomprise: receiving sensor data from at least one sensor that generatesbiophysical readings for a subject; by operation of a first neuralnetwork (NN), embedding the sensor data to generate embedded values; byoperation of a second NN, generating imputed embedded values in responseto the embedded values, the imputed embedded values including imputedvalues corresponding to one or more regions of the sensor data; andnormalizing the embedded imputed values to generate imputed values. Insome embodiments, the regions do not include data that is usable. Insome embodiments, receiving sensor data includes receiving data from afirst sensor and a second sensor different from the first sensor; andembedding the sensor data includes concatenating data from the first andsecond sensors. In some embodiments, the first sensor is a glucosemonitor. In some embodiments, the second sensor is a heart rate monitor.In some embodiments, the second NN comprises an autoencoder.

In another aspect, the present disclosure provides a system that cancomprise: at least one biophysical sensor that generates sensor datahaving missing regions where biophysical readings are determined to beinvalid or missing; a first neural network (NN) configured to embed datavalues from the at least one sensor to generate embedded values; asecond NN configured to generate imputed embedded values in response tothe embedded values, the imputed embedded values including imputedvalues corresponding to the missing regions of the sensor data; and anormalizing system configured to normalize the embedded imputed valuesto generate imputed values. In some embodiments, the at least one sensorincludes a first sensor and a second sensor different than the firstsensor; and the first NN is configured to embed sensor data from thefirst and second sensors in a same time period into single values. Insome embodiments, the first sensor is a glucose sensor. In someembodiments, the second sensor is a heart rate monitor. In someembodiments, the second NN comprises an autoencoder.

In another aspect, the present disclosure provides a method that cancomprise: receiving a validated data set and a query data set, each dataset including data values with labels; by operation of a neural network(NN), classifying the validated data set and query data sets with aprobabilistic classifier conditioned on the data set values and targetlabels; and generating a quality score based on a classification resultfor all data values of one data set. In some embodiments, the methodfurther comprises generating the query data set, including takingbiometric sensor readings with corresponding actions as labels. In someembodiments, the biometric sensor comprises a glucose meter. In someembodiments, the labels comprise food log data. In some embodiments, adistribution of the data values has the form p(X, Y, Z) where X is theinput distribution, Y is a categorical target of the probabilisticclassifier, and Z varies according to which data set the values belongto. In some embodiments, a classification of the probabilisticclassifier takes the form h(x)=p(z=1|x, Y=1), and z equals 0 if x isfrom the data set with validated labels and Z equals 1 if x is from thequery data set.

In another aspect, the present disclosure provides a system that cancomprise: a data storage system configured to store data sets includingdata values with labels, the data sets including at least a validateddata set and a query data set; and an electronic system in communicationwith the data storage system that includes at least one neural networkconfigured as a probabilistic classifier configured to classifying thevalidated data set and query data sets with conditioned on the data setvalues and target labels, and a quality section configured to examine aclassification value for all data values in the query or validated dataset and generate a quality value in response thereto. In someembodiments, the system further comprises: at least one biometric sensorconfigured to generate data values for the query data set. In someembodiments, the biometric sensor comprises a glucose meter. In someembodiments, the validated and query data sets include blood glucoselevels with food logs as labels. In some embodiments, a distribution ofthe data values has the form p(X, Y, Z) where X is the inputdistribution, Y is a categorical target of the probabilistic classifier,and Z varies according to which data set the values belong to. In someembodiments, a classification of the probabilistic classifier takes theform h(x)=p(X=1|x, Y=1), and Z equals 0 or 1 depending upon whether x isfrom the validated data set or query data set.

In another aspect, the present disclosure provides a method that cancomprise: storing biophysical sensor signals and logged behaviorcorresponding to the biophysical sensor signals in a data storagedevice, the stored data comprising training data; training a neuralnetwork on the training data to classify biophysical sensor signals asresulting in target behaviors; receiving input biophysical sensor data;and processing the input biophysical sensor data using the neuralnetwork to classify a target behavior that results from the inputbiophysical sensor data. In some embodiments, the biophysical sensorsignals include glucose sensor signals and the logged behavior includeslogged food data. In some embodiments, the biophysical sensor signalsinclude heart rate monitor signals and the logged behavior includeslogged food data. In some embodiments, the target behavior is predictedfood consumption. In some embodiments, the method comprises: acquiringthe input biophysical sensor signals with at least one sensor for asubject; transmitting the input biophysical sensor signals to the neuralnetwork; and transmitting the target behavior to a device of thesubject.

In another aspect, the present disclosure provides a system that cancomprise: a storage system configured to store training data comprisingtraining sensor data and corresponding behavior data; at least onebiophysical sensor configured to generate and transmit subject sensordata; and a behavior prediction system configured to receive the subjectsensor data and comprising at least one electronic system comprising aneural network trained as a classifier that classifies the subjectsensor data into a target behavior, the classifier trained with thetraining data. In some embodiments, the training data comprisingtraining sensor data from a plurality of different biophysical sensors;and the at least one biophysical sensor includes the plurality ofdifferent biophysical sensors. In some embodiments, the training sensordata includes glucose levels and the behavior data includes logged foodcorresponding to the glucose level; the at least one biophysical sensorincludes a glucose meter; and the target behavior is predicted foodingestion. In some embodiments, the training sensor data includes heartrate data and the behavior data includes logged food corresponding tothe heart rate data; the at least one biophysical sensor includes aheart rate monitor; and the target behavior is predicted food ingestion.In some embodiments, the system further comprises: the behaviorprediction system is further configured to transmit the target behavior;and a subject device configured to receive the target behavior.

In another aspect, the present disclosure provides a method that cancomprise: receiving and storing string data corresponding to adescription of a food item; applying the string data to a languageprocessor configured to determine nominative words and non-nominativewords from the string data; in response to the nominative words,querying an item database with the nominative words; in response tonon-nominative words, querying the item database with the non-nominativewords; and generating a list of query results in response to thequerying, the list of query results comprising recipes for the fooditems. In some embodiments, the method further comprises: the languageprocessor is further configured to determine nominative words asexplicit ingredients; and filtering the responses to the querying withthe explicit ingredients to generate the list of query results.

In another aspect, the present disclosure provides a system that cancomprise: a storage device configured to store a database comprisingdescriptions of food items; a language processing system comprising atleast one computing device configured to process text strings todetermine nominative and non-nominative words; a query system comprisingat least one computing device configured to apply first queries to thedatabase in response to the nominative words generated by the languageprocessing system and to apply second queries to the database inresponse to the non-nominative words to generate a list of query resultsin response to the queries, the list of query results comprising recipesfor the food items. In some embodiments, the language processing systemis further configured to determine nominative words as explicitingredients; and the query system is further configured to filterresponses to the first or second queries.

In another aspect, the present disclosure provides a method that cancomprise: receiving and storing first data comprising properties of anitem; receiving and storing second data comprising constituents of theitem ranked in order of prevalence in the item; determining theproperties for at least one of the ranked constituents in a database togenerate look-up data; determining at least one amount of the at leastone constituent in the item in response to the look-up data; and storingthe at least one amount of the at least one constituent as output data.In some embodiments, receiving and storing first data includes receivingnutrition information for a food item; and receiving and storing seconddata includes receiving ranked ingredient data for the food item. Insome embodiments, receiving and storing first and second data includescapturing and processing image data of a food label of the food item. Insome embodiments, the first data includes n properties; second dataincludes m constituents; determining the properties for each constituentincludes creating and storing an n×m matrix of constituents and theirproperties; and determining the amount of each constituent in the itemincludes solving a system of equations corresponding to y=Ax, where y isthe amount of an ingredient, x is a constituent and A is the matrix. Insome embodiments, determining the amount of each constituent includesapplying the matrix A to a neural network configured for linearregression analysis.

In another aspect, the present disclosure provides a system that cancomprise: a data pre-processing section coupled to receive first datacomprising properties of an item and second data comprising constituentsof the item having a ranked in order of prevalence in the item, andincluding a processing device configured to create a data structure thatrepresents properties for each constituent; and an analysis sectioncoupled to receive the data structure and including a processing deviceconfigured to determine determining the amount of each constituent inthe item. In some embodiments, the system further comprises: an inputdevice configured to capture the first and second data for the item. Insome embodiments, the input device comprises an image capture deviceconfigured to capture the image of a label for the item. In someembodiments, the first data comprises nutrition information of a fooditem and the second data comprises ingredients of the food item. In someembodiments, the first data includes n properties; second data includesm constituents; the data structure comprises a topological mapping ofconstituents and their properties; and the analysis section isconfigured to solve a system of equations corresponding to y=Ax, where yis the amount of an ingredient, x is a constituent and A is thetopological mapping. In some embodiments, the topological map is amatrix and the analysis section comprises a neural network configuredfor linear regression analysis.

In another aspect, the present disclosure provides a method that cancomprise: receiving and storing first data comprising properties of anitem; receiving and storing second data comprising constituents of theitem ranked in order of prevalence in the item; determining theproperties for at least one of the ranked constituents in a database togenerate look-up data; determining at least one amount of the at leastone constituent in the item in response to the look-up data; and storingthe at least one amount of each constituent as output data.

In another aspect, the present disclosure provides a method that cancomprise: training a word embedding system having a weighting matrixwith training data comprising string descriptions of items andproperties of the items to embed the string descriptions of items intoan embedded space weighted according to the properties of the items; andapplying an input string description of the item to the trained wordembedding system to infer an output word embedding weighted according tothe properties of the items. In some embodiments, training the wordembedding system includes training with food string descriptions withnutrition information as the properties. In some embodiments, applyingthe input string includes applying a string description of a food item.

In another aspect, the present disclosure provides a system that cancomprise: a storage system configured to store training data thatincludes string descriptions of items and properties of the items; aword embedding system using a neural network trained with the trainingdata to embed words of the string descriptions into an embedded spacewith a weighting derived from the properties of an item corresponding toone of the string descriptions; and an input configured to receive aninput string and apply it to the word embedding system to generate wordembeddings weighted according to the properties. In some embodiments,the training data includes word description of food items and nutritioninformation for the food items. In some embodiments, the input stringincludes a description of the food item and the generated wordembeddings are weighted according to the nutrition information.

In another aspect, the present disclosure provides a method that cancomprise: training a word embedding system having a weighting matrixwith training data comprising string description of items and propertiesof the items to embed the word string items into an embedded spaceweighted according to the properties of the corresponding item; andapplying an input string description of the item to the trained wordembedding system to infer output word embedding with the propertyweighting.

In another aspect, the present disclosure provides a system that cancomprise: a storage system configured to store training data thatincludes string descriptions of items and properties of the items; aneural network configured as word embedding system trained with thetraining data to embed words of the strings descriptions into anembedded space with a weighting derived from the properties of the itemcorresponding to the string description; and an input configured toreceive an input string and apply it to the trained word embeddingsystem to generate word embeddings weighted according to the properties.

In another aspect, the present disclosure provides a method that cancomprise (a) obtaining text-based descriptions of a plurality of fooditems and, for each of the plurality of food items, (i) nutrition dataand a glycemic value or (ii) nutrition data or a glycemic value; (b)generating embeddings of the text-based descriptions of the plurality offood items; (c) inferring, based at least on the embeddings, a glycemicvalue for each food item in the plurality of food items for which aglycemic value was not obtained and nutrition data for each food item inthe plurality of food items for which nutrition data was not obtained;(d) training a supervised machine learning algorithm on the nutritiondata and the glycemic values of the plurality of food items to predict aglycemic value of a given food item from nutrition data of the givenfood item. In some embodiments, the method comprises providing theglycemic value of the given food item to the supervised machine learningalgorithm to predict the glycemic value of the given food item. In someembodiments, the glycemic value is a glycemic index or a glycemic load.In some embodiments, (b) comprises applying an unsupervised learningalgorithm to the text-based descriptions of the plurality of food items.In some embodiments, the unsupervised learning algorithm is adimensionality reduction algorithm. In some embodiments, theunsupervised learning algorithm is an n-gram or bag-of-words model. Insome embodiments, the supervised machine learning algorithm is a deepneural network.

In another aspect, the present disclosure provides a system that cancomprise: a data storage system configured to store at least a firstdatabase and a second database, the first database includingdescriptions of first items with corresponding attributes, the seconddatabase including descriptions of second items with correspondingtarget values, at least some of the first items being different than thesecond items; an embedding system comprising at least a first computingdevice configured to merge the first and second databases to generatetraining data that includes merged item descriptions with correspondingattributes and target values; and at least a first inference systemcomprising a machine learning system trained with the training data toinfer target values from attributes. In some embodiments, thedescriptions of items comprise word descriptions. In some embodiments,the items are food items, the attributes are nutrition data of the fooditems, and the target values are glycemic response values. In someembodiments, the glycemic response values are selected from the groupof: a glycemic index and a glycemic load. In some embodiments, thesystem can further comprise a data capture section configured to acquirenutrition data with at least a subject device, and wherein the at leastfirst inference system is configured to infer a glycemic index valuefrom at least the acquired nutrition data. In some embodiments, thesystem can further comprise at least a second inference system that isconfigured to determine a blood glucose value of a subject in responseto at least glycemic response values of foods indicated as ingested bythe subject. In some embodiments, the embedding system comprises atleast one neural network configured to embed descriptions of first andsecond items into an embedded space.

In another aspect, the present disclosure provides a system that cancomprise: a data acquisition system configured to acquire attributevalues for items; and at least a first inference system configured toinfer target values from the acquired attribute values, the firstinference system including: at least one neural network trained withtraining data generated by embedding at least a first data set andsecond data set, the first data set including descriptions of items withcorresponding attributes, the second data set including descriptions ofitems with corresponding target values. In some embodiments, the systemfurther comprises a training agent configured to train the at least oneneural network with the training data. In some embodiments, the systemfurther comprises at least a second inference system configured to infera response for a subject from at least inferred target values. In someembodiments, the target values are glycemic response values for fooditems, and wherein the attribute values are nutrition values of the fooditems. In some embodiments, at least the attribute values are textvalues embedded into a vector space. In some embodiments, the systemfurther comprises an application server configured to transmit data toan application executed on a subject device in response to at least theinferred target values.

In another aspect, the present disclosure provides a method that cancomprise: training a neural network with time series training data of afirst modality and time series training data of a second modality tocreate a first model that generates time series data of the secondmodality from time series data of the first modality; training a secondmodel with the generated time series of the second modality, time seriestraining data of a third modality, and time series data of a fourthmodality to generate time series data of the fourth modality; until aconvergence condition is reached, iteratively testing the second modelon the time series data of the first modality and the time series dataof the third modality; and responsive to reaching the convergencecondition, predicting second modality data by testing the second modelwith data of the first modality. In some embodiments, the methodcomprises: acquiring the time series training data of the first modalitywith a first type sensor; and acquiring the time series training data ofthe second modality with a second type sensor. In some embodiments, thesecond type sensor is a glucose meter, and wherein the time series dataof the second modality includes glucose levels over time. In someembodiments, training the neural network to create the first modelincludes training with N sets of time series training data, and whereintraining the first model with the estimated time series training data ofthe first modality and time series training data of at least the thirdmodality includes training with M sets of time series data. In someembodiments, the method comprises testing the first model with the Nsets of time series data and the M sets of time series data and updatingthe first model in response to error values of the testing, and whereinthe trained first model is the first model with the smallest error. Insome embodiments, reaching a convergence condition includes calculatingan error value not greater than a threshold.

In another aspect, the present disclosure provides a system that cancomprise: an initial model section that includes a first model trainedto generate time series data of a second modality from time series dataof a first modality with M sets of training data; a training sectionthat includes: a second model derived from the first model andconfigured to generate time series data of at least a third modalityfrom at least time series data of a fourth modality with N sets oftraining data, and a testing section configured to test the second modelwith the M and N sets of training data, and update the second model inresponse to test error values; and an inference model that is the secondmodel with the lowest test error value, configured to infer time seriesdata of the second modality from time series data of the first modality.In some embodiments, the first model, the second model and the inferencemodel comprise neural networks. In some embodiments, the time seriesdata of the first and second modalities are biophysical sensor data. Insome embodiments, at least the time series data of the first and secondmodalities are glucose levels corresponding to glucose meters. In someembodiments, the third and fourth modalities are glucose levels. In someembodiments, the training section comprises: an inverse model that is aninverse of the first model and configured to generate estimated timeseries data of the first modality from the time series data of the thirdand a fourth modality; an estimator section configured to generatelinear parameters from the estimated time series data of the firstmodality and the time series data of the third modality; sectionconfigured to generate mapped time series data of the first modalityfrom time series data of the third modality using the linear parameters,wherein the second model is trained with the mapped time series data ofthe first modality.

In another aspect, the present disclosure provides a method that cancomprise: training a neural network with time series training data of afirst modality and time series training data of a second modality tocreate a first model that generates time series data of the secondmodality from time series data of the first modality; until aconvergence condition is reached: using a second model to generateestimated time series data of the first modality from a mixture of timeseries data from a third modality and a fourth modality, wherein thesecond model is initiated as an inverse model of the first model; usingthe estimated time series data of the first modality and time seriesdata of the third modality, training the second model to estimate linearfitting parameters; using the estimated linear fitting parameters togenerate analogous time series data of the first modality from the timeseries data of the third modality; linearly mapping the analogous timeseries data of the first modality to the time series data of the thirdmodality; training a third model using the linearly mapped analogoustime series data from the first modality mixture of time series data ofthe third modality and time series data of the fourth modality togenerate a mixture of time series data from the third modality and timeseries data from the fourth modality, wherein the third model is aninverse of the second model; modifying the second model to be an inversemodel of the third model; and evaluating whether the convergencecondition has been reached. In some embodiments, training the thirdmodel includes initializing the third model as the first model.

In another aspect, the present disclosure provides a method for traininga neural network to calibrate time series data. The method can comprisereceiving calibrated time series data for a biophysical response andcorresponding raw time series data for the biophysical response;training, on the calibrated time series data and the corresponding rawtime series data for the biophysical response, a neural network togenerate calibrated time series data, which training comprises updatingparameters of the neural network based on a difference between (i) anoutput of the neural network for a given raw time series and (ii) acorresponding calibrated times series; receiving raw input time seriesdata generated by a biophysical sensor; and generating calibrated timeseries data by applying the raw input time series data to the neuralnetwork. In some embodiments, the raw input time series data isgenerated by a glucose meter. In some embodiments, the neural network istrained to cancel drift present in the raw input time series data. Insome embodiments, the raw time series data and raw input time seriesdata are generated by glucose meters. In some embodiments, training theneural network further comprises domain specific feature engineering. Insome embodiments, training the neural network comprises unsupervisedtraining.

In another aspect, the present disclosure provides a method that cancomprise: building data structures from a plurality of data sets havingan ordering, the data structures including interval trees based on theordering; determining if any structures have missing intervals in theinterval tree; if a data structure has a missing interval, creating datafor the missing interval by imputing data values for the missinginterval; accessing the data structures by at least searching theinterval trees in response to query data; and forming a tabular datastructure from the accessed data values that includes a columnreflecting the ordering. In some embodiments, the data sets compriseactions ordered in time. In some embodiments, determining if any of thedata structures have missing intervals includes classifying datastructures into a first class if they have no missing intervals and asecond class if they have missing intervals. In some embodiments,accessing data values from the data structures includes an operationselected from the group consisting of: selecting a data structure for aquery operation; querying a region of a data structure dictated by theordering; joining query results; and merging overlapping regions ofdifferent data structures. In some embodiments, forming the tabular datastructure includes forming a dataframe from the accessed data values. Insome embodiments, forming the tabular data structure includes forming adataframe from the accessed data values. In some embodiments, the datasets comprise different subject events having an ordering, and whereinforming the tabular data structure includes forming a tabular datastructure that includes different subject events over a queried timeperiod. In some embodiments, at least one of the subject events is abiophysical response of the subject. In some embodiments, thebiophysical response is a glucose level of the subject.

In another aspect, the present disclosure provides a system that cancomprise: a data store configured to store tabular data sets, eachhaving data values with an ordering; and memory comprisingmachine-executable instructions that when executed by a processor causethe processor to perform operations comprising: create data structuresthat include interval trees based on the ordering, determining if any ofthe interval trees includes missing intervals, if an interval tree has amissing interval, imputing data for the missing interval, accessing datavalues from the data structures by at least searching the interval treesof the data structures in response to query data, and forming a tabulardata structure from the accessed data values that includes a columnreflecting the ordering. In some embodiments, the data store isconfigured to store tabular data sets having time or date columncorresponding to subject actions. In some embodiments, the processingsection is configured to execute an operation selected from the groupconsisting of: selecting a data structure query operation; querying aregion of a data structure dictated by the ordering; joining queryresults; and merging overlapping regions of structures. In someembodiments, the data store is configured to store tabular data setscomprising different subject events having an ordering, and wherein theprocessing section is configured to forming tabular data structures thatincludes different subject events over a queried time period. In someembodiments, at least one of the subject events is a biophysicalresponse of the subject. In some embodiments, the biophysical responseis a glucose level of the subject.

Another aspect of the present disclosure provides a non-transitorycomputer readable medium comprising machine executable code that, uponexecution by one or more computer processors, implements any of themethods above or elsewhere herein.

Another aspect of the present disclosure provides a system comprisingone or more computer processors and computer memory coupled thereto. Thecomputer memory comprises machine executable code that, upon executionby the one or more computer processors, implements any of the methodsabove or elsewhere herein.

Additional aspects and advantages of the present disclosure will becomereadily apparent to those skilled in this art from the followingdetailed description, wherein only illustrative embodiments of thepresent disclosure are shown and described. As will be realized, thepresent disclosure is capable of other and different embodiments, andits several details are capable of modifications in various obviousrespects, all without departing from the disclosure. Accordingly, thedrawings and description are to be regarded as illustrative in nature,and not as restrictive.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.To the extent publications and patents or patent applicationsincorporated by reference contradict the disclosure contained in thespecification, the specification is intended to supersede and/or takeprecedence over any such contradictory material.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings (also “Figure” and “FIG.” herein), of which:

FIG. 1 is a block diagram of a system according to an embodiment;

FIG. 2 is a block diagram of another system according to an embodiment;

FIG. 3 is a block diagram of a recommendation system according to anembodiment.

FIG. 4A is a block diagram of a recommendation system according toanother embodiment.

FIG. 4B is a block diagram of subject data input and output that can beused like that of FIG. 4A.

FIG. 4C is a block diagram biophysical reaction model according to anembodiment.

FIG. 4D is a block diagram of a predictor that can be included in asystem like of FIG. 4A.

FIG. 4E is a block diagram of an autoencoder (AE) and generativeadversarial network (GAN) that can be included in embodiments.

FIG. 5 is a block diagram of data prediction system according to anembodiment.

FIG. 6 is a block diagram of a biophysical prediction system accordingto an embodiment.

FIG. 7 is a block diagram of a system and method for encoding timeseries data to infer canonical actions of a subject.

FIG. 8 is block diagram of an evaluation system according to anembodiment.

FIG. 9 is block diagram of an evaluation system according to anotherembodiment.

FIG. 10 is block diagram of prediction system according to anembodiment.

FIG. 11 is a block diagram of a glucose level prediction systemaccording to an embodiment.

FIG. 12A is a flow diagram of a method for processing data objectsaccording to an embodiment;

FIG. 12B is code showing a method for decorating data objects for postprocess messaging according to an embodiment;

FIGS. 12C and 12D are diagrams showing the processing of data objectsaccording to an embodiment;

FIG. 13 is a flow diagram of a method according to an embodiment;

FIG. 14 is a flow diagram of a method of health management according toan embodiment;

FIG. 15 is a flow diagram of a method of coaching according to anembodiment;

FIGS. 16A to 16C are diagrams showing a data acquisition applicationaccording to an embodiment;

FIGS. 17A to 17F are diagrams showing a recommendation applicationaccording to an embodiment;

FIG. 18A is a block diagram showing a system and method for generating apersonalized biometric response according to an embodiment;

FIG. 18B is a block diagram showing a system and method for generating apersonalized glycemic response corresponding to a food source accordingto an embodiment;

FIG. 19A is a block diagram showing a system and method forautomatically predicting target values in response to attributes forsuch target values according to an embodiment;

FIGS. 19B and 19C are block diagrams showing a system and method forautomatically predicting a glycemic response as nutrients of a food itemare sequentially input to a system according to an embodiment;

FIG. 20A is a block diagram showing a system and method for imputingdata values for missing portions of a sensor data set according to anembodiment;

FIG. 20B is a block diagram showing a system and method for imputingdata values for one sensor based a model created with multiple sensorsaccording to an embodiment;

FIGS. 20C and 20D are diagrams showing one example of data imputationaccording to an embodiment;

FIG. 21A is a diagram of a data set quality determination system andmethod according to an embodiment;

FIG. 21B is diagram of a system and method for determining a quality ofdata sets that include sensor data labeled with logged behavior data;

FIG. 22A is a block diagram of a system and method for determining asubject behavior from sensor signals according to an embodiment;

FIG. 22B is a block diagram of a system and method for determining foodingestion in response to sensor signals according to an embodiment;

FIG. 23A is a block diagram of a system for determining the formula ofan item from a text description of the item according to an embodiment;

FIG. 23B is a block diagram of a system and method for determining thecomposition of a food item from a written description of the food itemaccording to an embodiment;

FIG. 24A is a block diagram of a system and method for determining theformula of an item from a text description of the item according to anembodiment;

FIG. 24B is a block diagram of a system and method for determining theamount of ingredients in a food item based on ranked ingredientsaccording to an embodiment. FIG. 24C is a diagram showing one example offood item data that can be acquired in an embodiment like that of FIG.24B;

FIGS. 25A and 25B are block diagrams of a system and method forembedding food string data into space weighted with nutritioninformation according to an embodiment;

FIG. 26 is a flow diagram of a method according to an embodimentaccording to an embodiment;

FIG. 27 is a flow diagram of a method according to another embodimentaccording to an embodiment;

FIG. 28 is a block diagram showing a system and method for creating amodel for predicting a target value from attribute values from data setsthat match targets with items and attributes with items according to anembodiment;

FIG. 29 is a block diagram showing a system and method for creating amodel for predicting glycemic values from nutrition facts from data setsthat match food items with nutrition facts and food items with glycemicvalues according to an embodiment;

FIG. 30 is a block diagram showing a system and method for creating amodel for predicting time series data, using time series training datasets, one or more of which may not have a high degree of accuracy;

FIG. 31 is a block diagram showing a system and method for creating amodel for predicting time series sensor data using time series sensordata of different modalities;

FIG. 32 is a block diagram showing a system and method for creating amodel for calibrating time series data;

FIG. 33A is a block diagram showing a system and method for creating adrift cancellation model for calibrating time series glucose data. FIGS.33B and 33C are diagrams of raw and calibrated time series data;

FIGS. 34 and 35 are block diagrams showing systems and methods forcreating interval tree like structures from tabular data sets to presentdata for events across the data sets according to embodiments;

FIGS. 36A to 36E are diagrams showing data sets and outputs for a systemlike that shown in FIG. 34 or 35; and

FIG. 37 shows a computer system that is programmed or otherwiseconfigured to implement methods provided herein.

DETAILED DESCRIPTION

While various embodiments of the invention have been shown and describedherein, it will be obvious to those skilled in the art that suchembodiments are provided by way of example only. Numerous variations,changes, and substitutions may occur to those skilled in the art withoutdeparting from the invention. It should be understood that variousalternatives to the embodiments of the invention described herein may beemployed.

Whenever the term “at least,” “greater than,” or “greater than or equalto” precedes the first numerical value in a series of two or morenumerical values, the term “at least,” “greater than” or “greater thanor equal to” applies to each of the numerical values in that series ofnumerical values. For example, greater than or equal to 1, 2, or 3 isequivalent to greater than or equal to 1, greater than or equal to 2, orgreater than or equal to 3.

Whenever the term “no more than,” “less than,” or “less than or equalto” precedes the first numerical value in a series of two or morenumerical values, the term “no more than,” “less than,” or “less than orequal to” applies to each of the numerical values in that series ofnumerical values. For example, less than or equal to 3, 2, or 1 isequivalent to less than or equal to 3, less than or equal to 2, or lessthan or equal to 1.

The present disclosure provides systems and methods that can acquiresensor and other data that records subject actions and utilizereinforcement learning to predict a subject response. The prediction canbe a prediction of a biophysical response and/or behavioral response.Embodiments can utilize custom variational encoding to model subjectactions and responses. In some embodiments, the systems described hereincan generate a recommendation for a subject based on a reward functionand the subject's historical actions.

FIG. 1 shows a system 100 according to an embodiment. The system 100 caninclude one or more of any of the following: machine learning (ML)servers 102, application servers 104, a data store 106, data sources108-0 to 108-2, and subject device 110. The data sources 108-0 to 108-2,ML and application servers (102, 104) and subject devices 110 can be incommunication with one another through a communication network 112. Thecommunication network 112 can be wired or wireless. For example, thecommunication network 112 can be a Bluetooth network, a Wi-Fi network, alocal area network, a wide area network, a cellular network, or thelike. In some cases, the communication network 112 can be the Internet.

The ML servers 102 can include appropriately-programmed hardware forimplementing the various ML systems and functions described herein. Thehardware can be general-purpose processors, graphics processing units(GPUs), application-specific integrated circuit (ASIC), or machinelearning accelerators, to name a few examples. The ML servers 102 canimplement artificial neural networks (ANN) of various architectures aswill be described herein. Such ANNs can perform various functions,including learning and inference operations on data received from datasources 116, 118, and 120 as well as other data residing on the datestore 122. The ANNs can autoencoders (AEs), generative adversarialnetworks (GANs), long short-term memory networks (LSTMs), convolutionalneural networks (CNNs), reinforcement learning (RL) algorithms, and anyother artificial neural network (ANN) or related architecture suitablefor the systems and methods described herein.

In general, the ML algorithms implemented on the ML servers 102 can beused to predict a subject's biophysical response (e.g., a glucoseresponse) or make a recommendation (e.g., a diet or physical activityrecommendation) that is configured to alter or maintain an aspect of thesubject's health (e.g., glucose level). The ML algorithms can besupervised learning algorithms, semi-supervised learning algorithms,unsupervised learning algorithms, reinforcement learning algorithms, orthe like.

A supervised ML algorithm can be trained using labeled training inputs,i.e., training inputs with known outputs. The training inputs can beprovided to an untrained or partially trained version of the MLalgorithm to generate a predicted output. The predicted output can becompared to the known output, and if there is a difference, theparameters of the ML algorithm can be updated. A semi-supervised MLalgorithm can be trained using a large number of unlabeled traininginputs and a small number of labeled training inputs. An unsupervised MLalgorithm, e.g., a clustering or dimensionality reduction algorithm, canfind previously unknown patterns in data sets without pre-existinglabels.

A reinforcement learning algorithm may seek an optimal solution to aproblem by balancing exploration of uncharted territory withexploitation of current knowledge. In reinforcement learning, labeledinput-output pairs need not be used. Instead, an agent (e.g., an MLalgorithm) can choose an action from a set of available actions. Theaction may result in a new environmental state. The new environmentalstate may have a reward associated with it, and the reward may bepositive or negative depending on whether the new state is better orworse than the previous state. The goal of the agent may be to collectas much reward as possible, e.g., optimize a subject's glucose level.The set of available actions from which the agent can choose may be aprobability distribution of actions. The probability distribution may beadjusted as the agent receives rewards. That is, actions that result innegative rewards may be slowly filtered out of the probabilitydistribution, while actions that result in positive rewards may beemphasized in the probability distribution. In the context ofbiophysical responses, the state may be a subject's glucose level, andthe reward function may reward recommendations (e.g., medical, diet, orphysical activity recommendations) that maintain or achieve a normalglucose level.

The ML algorithms used herein may be neural networks. Neural networkscan employ multiple layers of operations to predict one or more outputs,e.g., the glucose level of a subject. Neural networks can include one ormore hidden layers situated between an input layer and an output layer.The output of each layer can be used as input to another layer, e.g.,the next hidden layer or the output layer. Each layer of a neuralnetwork can specify one or more transformation operations to beperformed on input to the layer. Such transformation operations may bereferred to as neurons. The output of a particular neuron can be aweighted sum of the inputs to the neuron, adjusted with a bias andmultiplied by an activation function, e.g., a rectified linear unit(ReLU) or a sigmoid function.

Training a neural network can involve providing inputs to the untrainedneural network to generate predicted outputs, comparing the predictedoutputs to expected outputs, and updating the algorithm's weights andbiases to account for the difference between the predicted outputs andthe expected outputs. Specifically, a cost function can be used tocalculate a difference between the predicted outputs and the expectedoutputs. By computing the derivative of the cost function with respectto the weights and biases of the network, the weights and biases can beiteratively adjusted over multiple cycles to minimize the cost function.Training may be complete when the predicted outputs satisfy aconvergence condition, e.g., a small magnitude of calculated cost asdetermined by the cost function.

Examples of neural networks include CNNs, recurrent neural networks(RNNs) (e.g., LSTMs), and others. CNNs are neural networks in whichneurons in some layers, called convolutional layers, receive pixels fromonly small portions of the input data set. These small portions may bereferred to as the neurons' receptive fields. Each neuron in such aconvolutional layer can have the same weights. In this way, theconvolutional layer can detect features in any portion of the input dataset. CNNS may also have pooling layers that combine the outputs ofneuron clusters in convolutional layers and fully-connected layers thatare similar to traditional layers in a feed-forward neural network. Insome cases, CNNs may be used to detect objects in any portion of animage or video.

RNNs, meanwhile, are neural networks with cyclical connections that canencode dependencies in time-series data, e.g., continuous glucosemonitoring data, An RNN can include an input layer that is configured toreceive a sequence of time-series inputs. An RNN can also include one ormore hidden recurrent layers that maintain a state. At each time step,each hidden recurrent layer can compute an output and a next state forthe layer. The next state can depend on the previous state and thecurrent input. The state can be maintained across time steps and cancapture dependencies in the input sequence. Such an RNN can be used toencode times-series features of a subject's glucose levels, for example.

One example of an RNN is an LSTM, which can be made of LSTM units. AnLSTM unit can be made of a cell, an input gate, an output gate, and aforget gate. The cell can be responsible for keeping track of thedependencies between the elements in the input sequence. The input gatecan control the extent to which a new value flows into the cell, theforget gate can control the extent to which a value remains in the cell,and the output gate can control the extent to which the value in thecell is used to compute the output activation of the LSTM unit. Theactivation function of the LSTM gate can be the logistic function.

The ML algorithms used here may alternatively or additionally be GANs. AGAN can include a generative network and a discriminative network. Thegenerative network can generate candidate simulations while thediscriminatory network can evaluate the candidate simulations. The goalof the discriminatory network may be to distinguish between a simulationand a true data distribution, while the goal of the generative networkmay be to increase the error rate of the discriminatory network.Backpropagation can be applied to both networks so that the generativenetwork produces better simulations, while the discriminative networkbecomes more skilled at flagging simulations.

The ML algorithms used herein may alternatively or additionally be AEs.AEs can have an encoder that is configured to generate areduced-dimension representation of an input and a decoder that is theconfigured to reconstruct the input from the reduced-dimensionrepresentation. An AE can be trained by comparing the input to theoutput and adjusting the weights of the encoder and decoder accordingly.One of the main purposes of AEs is to extract features from data. An AEcan be used to detect anomalous data, e.g., data that is different thanthe training data.

In some embodiments, the ML servers 102 can include reinforcementlearning (RL) agents 114 that can operate in response to inputs fromdata sources 108-0 to -2 to generate suggested actions based on adesired reward function. Such suggested actions can be provided to auser device 110 by operation of a ML or application server (102, 104).Subject responses and behavior as recorded by data sources (108-0 to -2)can be encoded into a latent space with custom variational encoding 116to model and predict subject responses. In particular embodiments, MLservers 102 can include a personalized blood glucose predictor 118 forpredicting subject blood glucose levels, and recommendations generatedby RL agents 114 can be actions to help maintain blood glucose levelspredetermined-levels.

In other embodiments, the ML servers 102 can include training datageneration systems and feature prediction systems. The training datageneration systems can use ML processes to generate training data totrain the feature prediction system, which can also use ML processes. Insome embodiments, the training data generation systems can embeddescriptive data to enable a targeted feature to be inferred from suchdescriptive data. In this disclosure, the word “embed” or “embedding”may refer to a process by which words or phrases, e.g., text-baseddescriptions of food items, are mapped to vectors of real numbers. Theresulting vector space may have a lower dimension than the input (i.e.,the words and phrases). As such, embedding may be considered adimensionality reduction technique. The feature inference system can betrained to infer a target feature, e.g., a glycemic index of the fooditems, from the embeddings.

The application server 104 can interact with one or more applicationsrunning on the subject device 110. In some embodiments, data from datasources (108-0 to -2) can be acquired via one or more applications onthe subject device 110 and provided to application server 104.Application server 104 can communicate with subject device 110 accordingto any suitable secure network protocol. The application server 104 canreside on the same or different physical server as the ML servers 102.The application server 104 can relay subject data from the subjectdevice 110 to the ML server 102.

The data store 106 can store data for system 100. In some embodiments,data store 106 can store data received from data sources (108-0 to -2)(e.g., data from one or more subjects) as well as other data setsacquired by third parties. Data store 106 can also store various othertypes of data, including configuration data for configuring operationsfor ML servers 102. The data store 106 can take any suitable form,including one or more network-attached storage systems. In someembodiments, all or a portion of the data store 106 can be integratedwith any of the ML or application servers (102, 104).

In some embodiments, data for data sources (108-0 to -2) can begenerated by sensors or can be logged data provided by subjects. In FIG.1, data source 108-0 can correspond to a first type sensor 120-0 (e.g.,a heart rate monitor), data source 108-1 can correspond to a second typesensor 120-1 (e.g., a continuous glucose monitor), and data source 108-2can correspond to logged data 122 provided by a subject. Logged data 122can include text data or image data (e.g., text data or image datadescribing or defining nutritional information of a food).

The second type sensor 120-1 and the logged data 122 can be “indirect”data sources in that data from such sources can be used to infer otherdata. For example, data from the second type sensor 120-1 and the loggeddata 122 can be used to infer data from the first type sensor 120-0,which may be considered a “direct” data source. In some embodiments,logged data 122 can be processed to infer a biophysical responsedifferent from the response(s) that the second type sensor 120-1 recordsor detects. In some embodiments, both direct and indirect data can beused to train and calibrate biophysical models, however, direct data maynot be used in inference operations in such embodiments. Instead, onlythe indirect data sources may be used during inference. In someembodiments, the first type sensor 120-0 can be a sensor that is moredifficult to employ than the second type sensor 120-1. The sensors canrecord data and transmit the data to the subject device 110 over a localnetwork (e.g., Bluetooth network). The subject device 110 can thentransmit the data to one or more servers (e.g., ML servers 102 orapplication server 104). In addition or alternatively, such sensors canalso transmit such data to one or more servers (102, 104) without asubject device (e.g., directly, or via one or more intermediatedevices).

In some embodiments, the first type sensor 120-0 can be a continuousglucose monitor (CGM), which can track a glucose level of a subject. Thesecond type sensor 120-1 can be heart rate monitor (HRM) which can tracka subject's heart rate. Logged data 122 can be subject nutrition data.In some embodiments, an application running on the subject device 110can acquire the logged data 122. In some embodiments, the applicationcan capture an image. The image can be, for example, an image of anutrition label on a pre-packaged food item, an image of a barcode thatencodes nutritional information for a particular food, one or moreactual food items (e.g., a piece of fruit or a full meal), or the like.ML algorithm on the ML servers 102 can infer nutrition values from theimages and, using such nutrition values, infer the glucose response ofthe subject. The image can be an image of text (e.g., labels 122-1)which can be subject to optical character recognition to generatecomputer-readable text, and such text can be applied to an inferenceengine.

While FIG. 1 shows particular data sources for particular biophysicalmodeling and prediction, embodiments can include any other suitablesensor applications, particularly those applications having a“difficult” sensor (e.g., a direct sensor) that is more difficult,complex, or expensive to implement than one or more other “easy” sensorsand/or subject data logging. Data from a difficult sensor can be used totrain ML models that can infer a subject response from data from easysensors and/or subject data logging as inputs.

The subject device 110 can be any suitable device, including but notlimited to, a smart phone, personal computer, wearable device, or tabletcomputing device. The subject device 110 can include one or moreapplications that can communicate with application server 104 to providedata to, and receive data from, biophysical models residing on MLservers 102. In some embodiments, the subject device 110 can be anintermediary for any of data sources (108-0 to -2). The communicationnetwork 112 can be any suitable network, including a local area network,wide area network, or the internet, for example.

Referring to FIG. 2, a system 200 according to another embodiment isshown in a block diagram. The system 200 can include data source inputs208-0, 208-1, 208-2, a subject data capture portion 224, a storageportion 206, a data pre-processing portion 226, a ML services portion202, and an application services portion 204. Data source inputs (208-0,208-1, 208-2) can provide data for learning operations in ML servicesportion 202 that create biophysical models for a subject. Any or all ofdata source inputs (208-0, 208-1, 208-2) can provide data for inferenceoperations executed on models resident in ML services portion 204. Invery particular embodiments, data source inputs (208-0, 208-1, 208-2)can include any of the sensors and/or subject data logging describedherein or equivalents.

Data store portion 206 can include subject data storage 206-0 as well asnon-subject data storage 206-1. Subject data storage 206-0 can be datafor particular subjects for which ML models have been created or arebeing created. Non-subject data storage 206-1 can include data derivedfrom other sources that can be used for other purposes such as trainingand creating models. Such data can include, but is not limited to, datafrom non-subjects, such as participants in third-party studies.

A data pre-processing portion 226 can process data from data storeportion 206 Data pre-processing portion 226 can include instructionsexecutable by a processor to place data into particular formats forprocessing by ML services portion 202.

ML services portion 202 can include computing systems configured tocreate ML models and architectures through supervised and/orunsupervised learning with any of data source inputs 208-0, 208-1,208-2. In addition, ML services portion 202 can include ML models and/orarchitectures that generate inference results based on any of datasource inputs 208-0, 208-1, 208-2. ML services portion 202 can includesingle computing devices that include ANNs of the various architecturesdescribed herein, as well as ANNs distributed over networks. As in thecase of FIG. 1, ML services portion 202 can include AEs, GANs, LSTMs,CNNs, RL algorithms, and any other suitable ANN or other statisticallearning agent, and related architectures. In some embodiments, MLservices portion 202 can include reinforcement learning agents 214,custom variable encoders 216, and one or more networks configured topredict a reaction 218 customized to a user based on any of data sourceinputs (208-0 to -2).

Application services 204 can access models or other networks resident inML services portion 202 to provide data for one or more subjectapplications 228 resident on a subject device 210. Applications 228 canutilize model/network outputs to provide information to subjects. Insome embodiments, application services portion 228 can providerecommended actions for subjects based on subject responses predicted bymodels/networks in ML services portion 202. In some embodiments,application services portion 228 can recommend subject actions based onpredicted glucose levels of subjects and subjects recorded activities.The recommended actions can be diet-related, physical activity-related,or the like. While application services portion 204 can serviceapplications 228 running on subject devices 110, in other embodimentsapplication services 204 can execute applications and provide (e.g.,push) data to other services (e.g., email, text, social network, etc.).

Referring still to FIG. 2, example operations performed by the system200 will now be described. The sensors 208 can acquire two or moredifferent types of data. In some cases, one type of data may be an inputfeature or attribute and another type of data may be a target output(e.g., an output to be predicted by an ML algorithm). Both types of datamay be associated with text-based descriptions. The ML services portion202 can generate embeddings of the text-based descriptions using anembedding function. Such embeddings can be used to infer the inputfeatures or attributes or the target output. The inferred values can beused to train the system 200 to predict the target output from the inputfeature or attribute.

For example, the embedding function can generate embeddings ofdescriptions of food items. Thereafter, the embeddings and correspondingglycemic values for such food items (which serve as labels) can be usedto train the inference system (e.g., a supervised machine learningalgorithm such as an ANN) to predict glycemic values using only standardnutrition data of the food items, which may be more readily availablethan glycemic values.

While embodiments can include numerous systems and methods for modelingand predicting subject responses, some embodiments can include systemsand methods for making personalized recommendations for a subject, basedon predicted reactions of a subject.

FIG. 3 is a block diagram of a recommendation system 300, according toan embodiment. The system 300 can generate recommendations for a subject330. The system 300 can include a subject reaction model 318, rewardfunctions 332-0/1, and an RL section 314. The subject reaction model 318can be a personalized model for the subject 330. The system 300 caninclude a high frequency loop 336 and a low frequency loop 338.

The high frequency loop 336 can include the RL section 314, the subjectreaction model 318, and the reward function 332-0. The RL section 314can be an ML model. For example, the RL section 314 can be a neuralnetwork. The RL section 314 can initially be configured with randomweights or parameters. The RL section 314 can be configured to generatea recommendation 340-0 for a subject 330. The recommendation 340-0 canbe a diet, physical activity, sleep, hydration, or stress releaserecommendation, for example. Based on recommendation 340-0 and subjectreaction 334, subject reaction model 318 can generate a predictedsubject reaction 344. The predicted subject reaction 344 can be thesubject's predicted reaction to the recommendation 340-0. The rewardfunction 332-0 can process the predicted subject reaction 340-0 togenerate a reward for the RL section 314. The reward function 332-0 cangenerate a positive reward if the predicted subject reaction 340-0 isbeneficial to a particular health measurement of interest (e.g., thesubject's glucose level) and a negative reward if the predicted subjectreaction is detrimental to the health measurement of interest. Theweights or parameters of the RL section 314 can be adjusted to accountfor the award. In general, the RL section 314 may iteratively adjust itsweights or parameters to maximize the reward it receives. Such actionscan continue until high frequency loop 336 arrives at a particularsubject recommendation 340-1 (for example an optimal recommendation)which can be issued to subject 330. Subject recommendation 340-1 can begenerated according to various criteria. For example, a reward functionvalue, number of iterations, or amount of time passed, to name only afew.

The low frequency loop 338 can use a subject's actual response torecommendations to generate new recommendations. The low frequency loop338 can include recommendation 340-1 (arrived at by RL section 314according to predetermined criteria), reward function 332-1, and RLsection 314. The subject actual response 334 to the low frequency (e.g.,optimal) recommendation 334-1 can be evaluated by a reward function332-1, to generate inputs to RL section 314. RL section 314 can seek tomaximize a reward to generate a recommendation 340-0 (which may beincluded in high frequency loop 336).

While recommendation systems as described herein can be implemented invarious applications, some embodiments can model a subject's biophysicalresponse to provide recommendations for achieving a goal, such asimproved health. Such an embodiment is shown in FIG. 4.

FIG. 4A shows a recommendation system 400 according to anotherembodiment. The recommendation system 400 can be an implementation ofthe system 300 of FIG. 3. Recommendation system 400 can provide healthrelated recommendations for a subject based on biophysical responses.Biophysical responses can include any responses described herein (e.g.,glucose response, insulin, weight). In some embodiments, recommendationscan include physical activities and/or nutrition suggestions based onbiophysical sensor readings and/or other data logged by a subject. Inparticular embodiments, an encoded biophysical response 434 can includeheart rate monitor data and logged food data encoded into a latentspace.

In FIG. 4A, subject sensor and/or data logging 430 can provide encodedsubject responses 434 and historical data for a subject 442 to a system400. In response, a system 400 can provide an encoded recommendation440-1.

The system 400 can have a high frequency loop 436. The high frequencyloop 436 can have a RL section 414, a subject biophysical reaction model418, and a reward function 432-0. The RL section 414 can generate arecommendation 440-0. The recommendation 440-0 can be a recommendationto eat a particular food, participate in a physical activity, or thelike. Subject biophysical reaction mode 418 can receive therecommendation 440-0, as well as subject historical data 442 and encodedsubject biophysical responses 434 for a subject. In response, subjectbiophysical reaction mode 418 can generate a predicted subject reaction444. The predicted subject reaction 444 can be evaluated by rewardfunction 432-0. Reward function 432-0 can base its evaluation on ahealth-related outcome. In some embodiments, the health-related outcomecan be a function of the blood glucose level of a subject. The resultingoutput of the reward function 432-0 output can be provided to RL section414. The weights of the RL section 414 can be adjusted based on theoutput of the reward functions 432-0. High frequency loop 436 cancontinue until a predetermined point at which RL section 414 can issue acurrent, low frequency recommendation 438 to a subject. Such apredetermined point can be based on some quantitative value of rewardfunction (e.g., convergence, optimality), or number of iterations, ortime-based periodicity, or some combination thereof, to name a fewexamples.

The low frequency loop 438 can include reward function 432-1 and RLsection 414. Encoded subject biophysical responses 434 can be applied toreward function 432-1. As in the case of reward function 432-0, rewardfunction 432-1 can evaluate responses 434 on a health-related outcome.In some embodiments, reward function 432-1 can have the same or similarreward states as reward function 432-0. The resulting output of rewardfunction 432-0 can be provided to RL section 414. As noted above, RLsection 414 can receive more frequent reward function evaluations fromhigh frequency loop 436.

FIG. 4B is a diagram showing subject sensor and/or data logging 430according to an embodiment and can be one implementation of that shownin FIG. 4A. Subject sensor and other data 446 can be received from anysuitable source as described herein and equivalents. Such data 446 canbe received in processed form, including encoded form, however in FIG.4B data 446 is received in unencoded form.

Subject sensor and other data 430 can be encoded by an encoder 448 togenerate encoded subject biophysical response 434 for use by a rewardfunction and/or subject biophysical reaction model. Encoded data 430 canalso be stored or further encoded in data history 450, which can beaccessed to acquire subject historical data 442. In the embodimentshown, low frequency encoded recommendations 440-1, which can bereceived from an RL section, can be decoded by a decoder 452 to generateunencoded recommendations 440-3 that can be presented for a subject.

FIG. 4C shows a subject biophysical reaction model 418 according to anembodiment. The subject biophysical reaction model 418 can include apredictor 454, adherence model 456, decoder 458, switch function 460,body model 462, parameter estimator 464, and an AE and GAN 466. Usingencoded subject historical data 442, the predictor 454 can generate apredicted subject reaction 468 (in encoded form). The predictor 454 canalso provide latent data 470 regarding a subject's actions to body model462 and adherence model 456.

In response to a high frequency recommendation 440-0, adherence model456 can provide an adherence output 472, that indicates to what extent asubject's actions follow the recommendation 440-0 (e.g. if the subjecteats a recommended food or participates in a recommended physicalactivity). Adherence output 472 can be in encoded form.

A switch function 460 can selectively apply a predicted subject reaction468 or adherence output 472, when a recommendation is simulated, as aninput to a body model 462. In the embodiment shown, a body model 462 canoperate in response to unencoded data and so a decoder 458 can translateinputs from switch function 460 from a latent space into unencodedreaction data 474.

Unencoded reaction data 474 (which can be derived from a predictedsubject reaction or adherence model evaluation) can be applied to a bodymodel 462 to generate a simulated subject biophysical reaction 476. Thebody model can be a personalized learned model for a specific subject.During inference, inputs (e.g., food consumption and heart rate) of theuser and latent states of the user can be used predict the physicalobservables of the user (e.g., glucose values). The simulated subjectbiophysical reaction 476 can be applied to an AE and GAN 466 to producea more enhanced predicted subject biophysical reaction 444. The AE andGAN 466 can receive the output of the body model 462 which can be asimulated glucose curve and modify the simulated glucose curve to ensureit resembles real glucose values. In other words, the AE and GAN 466 canadd deep learning on top of simulated observables to assure that thesimulated and real glucose values will not be distinguishable.

Referring still to FIG. 4C, while predicted subject biophysicalreactions 476 are generated as described above, a body model 462 can beupdated in response to a subject actual biophysical response representedby encoded subject biophysical response 434. In the embodiment shown, inresponse to a subject's biophysical response 434, a model parameterestimator 464 can update parameters within body model 462 to seekconvergence between simulated subject biophysical reactions 476 andactual subject biophysical response (e.g., 434). For example, theparameters of the body model may be updated via a supervised learningprocess, during which the encoded subject biophysical responses 434 areused as training data labels.

FIG. 4D shows a predictor 454 that can be included in embodiments,including that of FIG. 4C. Predictor 454 can include an encoder 448 anddecoder 458. In some embodiments, encoder 448 can be an autoencoder. Inparticular embodiments, encoder 448 can be a custom variational encoderas described herein, or an equivalent. Encoder 448 can be trained topredict subject reactions from subject historical data by mappinghistorical data into latent space 478. In response to subject historicaldata 442, encoder 448 can infer predicted subject reactions. Suchreactions can be decoded by decoder 458 to generate a predicted subjectreaction 468. In some embodiments, encoder 448 can be trained to mapinputs that satisfy the preconditions of (1) minimizing reconstructionlosses when decoded by decoder 458 and (2) preserving predeterminedminimum distance metrics in the latent space 478.

FIG. 4E shows an AE and GAN 466 that can be included in embodiments,including that of FIG. 4C. AE and GAN 466 can include an encoder 448, agenerator 482, and discriminator 484. Encoder 448 can map simulatedsubject biophysical reactions 476 into a latent space 480. Outputs fromencoder 448 can be applied to generator 482. Generator 482, incombination with discriminator 484, can form a GAN. Generator 482 canoutput predicted subject biophysical reactions 444. Discriminator 484can be trained with actual biophysical reactions and provide feedback togenerator 482 so that that predicted biophysical reactions 444 may moreclosely follow those of actual subjects.

Embodiments can also include systems and methods for predicting dataseries from past heterogenous data sources. FIG. 5 is a block diagram ofa data prediction system and method 500 according to an embodiment. Thedata prediction system 500 can include various different data sources508-0 to 508-n (e.g., food, heart rate, sleep, calendar, etc.), AEtemporal convolutional neural networks (CNNs) 548-0 to 548-n,corresponding past data sources 508-0′ to 508-n′, temporal CNN encoders588-0 to 588-n, and a concatenate/forecast section 590. Any CNN can bereplaced by an RNN type network, such as an LSTM as but one example.

Each of the AE temporal CNNs (548-0 to 548-n) can receive and be trainedwith data from a different data source (508-0 to 508-n). In response tosuch training, AE temporal CNNs (548-0 to 548-n) can provide seedvalues, which are a set of priors and initial states, (586-0 to 586-n)to the temporal CNN encoding (588-0 to 588-n). Seed values (586-0 to586-n) can configure their corresponding temporal CNN encoding (588-0 to588-n) which can have a same encoding architecture as AE temporal CNNs(548-0 to 548-n). Temporal CNN encoding (588-0 to 588-n) can thenreceive past data source values (508-0′ to 508-n′), which are of thesame type used to generate the seed values (586-0 to 596-n). Such pastdata source values (508-0′ to 508-n′) can then be encoded to generateencoded values 592-0 to 592-n.

Encoded values 592-0 to 592-n, can be applied to an ANN withinconcatenate/forecast section 590, which can generate predicted values asdata forecast 594. Forecast 594 can represent predictions for any or allof values represented by data sources 508-0 to -n.

FIG. 6 shows a block diagram of a system 600 for predicting one type ofbiophysical response from multiple, different biophysical responses.System 600 can be one particular implementation of that shown in FIG. 5.In the particular embodiment shown, system 600 can predict a futureglucose level from heterogenous data sources of a CGM, HRM and foodlogging.

The system 600 can receive data from a CGM source 608-0, an HR source608-1 and a food logging source 608-2. Data from the CGM source 608-0can be received by an AE temporal CNN 648-0, which can generate seedvalues 686-0, which are a set of priors and initial states. Data fromthe HR source 608-1 can be received by an AE temporal CNN 648-1, whichcan generate seed values 686-1. Data from the food logging source 608-2can be received by an AE temporal CNN 648-2, which can generate seedvalues 686-2.

A temporal CNN encoding 688-0 can be seeded with seed values 686-0 andencode past data from a CGM source 608-0′ to generate encoded CGM values692-0. Similarly, temporal CNN encoding 688-1 can be seeded with seedvalues 686-1 and encode past data from a HR source 608-1′ to generateencoded HR values 692-1, and temporal CNN encoding 688-2 can be seededwith seed values 686-2 and encode past data from food logging 608-2′ togenerate encoded food logging values 692-2.

Encoded CGM values 692-0, encoded HR values 692-1 and encoded foodlogging values 692-2 can be applied to concatenate/forecast section 690,which can be configured to generate predicted CGM data (CGM forecast694). Concatenate/forecast section 690 can be any suitable architectureconfigured and/or trained to forecast CGM values from the encoded CGM,HR and food logging values (692-0 to -2). For example, theconcatenate/forecast section 690 can have a machine learning layer andan output layer (e.g., a softmax layer).

While embodiments can include systems and methods for generatingrecommendations or forecasting responses based on sensor data, otherembodiments can include systems and methods that can classify humanbehavior into discrete actions. Such discrete actions can be used toarrive at recommendations or other operations to effect subjectbehavior.

FIG. 7 is a block diagram of a system 700 for classifying behavior of asubject (e.g., eating, physical, activity, sleeping, etc.) according toan embodiment. The system 700 can include an encoder 748, an orthogonalsignal selection 796, a decoder 782, a mapping section 795, anoptionally, a personalization filter 793.

The encoder 748 can receive time series data from different data sourcesshown as 708-0 to 708-n. The data sources (708-0 to -n) can each providedata from one or more subjects. According to some embodiments, the datasources (708-0 to -n) can include sensors that can provide data forphysiological responses of a subject or subjects. Data sources (708-0 to-n) can include but are not limited to a glucose monitors (e.g., CGM),HR monitors, food consumption data (e.g., food logging), sleep state,accelerometer readings, calendar data, IR geographic data (e.g., GPS).While time series data can be encoded in any suitable timeframe for thedesired encoding result, in some embodiments, time series data can beencoded in increments of no more than about an hour, no more than about30 minutes, or no more than about 15 minutes. In some cases, theincrements can be longer than an hour.

The encoder 748 can be trained to maintain a predetermined metricdistance in a resulting latent space 780. This can include implementingdistance metrics 799 that can seek to cluster values in latent space,while maintaining separation between clusters. In some embodiments, theencoder 748 can be a temporal CNN variational AE. Time series data fromsensors (708-0 to -n) can take any suitable form, but in someembodiments can include consecutive 15-minute sections of sensor data.

Orthogonal vector selection 796 can select a set of orthogonal vectorssuitable to the particular type of encoder 748. Vectors can be selectedbased on a resulting latent space 780. Decoder 782 can include adecoding ANN or other network corresponding to the encoder 748. Decoder782 can receive inputs from orthogonal signal selection 796 and generatean output that represents a set of statistically common behaviors.Optionally, such behaviors can be filtered by a personalization filter797. For example, common behaviors can be selected or eliminated basedon relevance to the subject(s).

Mapping section 795 can map the sensor space results to approximateactions of subject(s). Such actions can present “canonical” discreteactions 793 for use in recommendations, control suggestions, etc.

While embodiments can include systems and methods for predicting asubject response with subject models, embodiments can also includemethods and systems for monitoring and diagnosing such models. A systemcan be in communication with various models that infer subjectresponses. The results of such models can be compared with actualsubject responses that serve as reference values.

FIG. 8 shows an evaluation system 800 according to an embodiment. Thesystem 800 can implement a self-consistent model to evaluate and train acomplex model that consists of various machine-learning blocks. Theaccuracy of a complex model is affected by each one of themachine-learning blocks in it. To evaluate each one of these blocks, thesystem 800 can compare the outputs and introduce the main source oferror and then the adjust the models can update the lossy ones. Thesystem 800 can include a sensor 818 (which can serve as data source),other data sources 808-1 and 808-2, data source models (891-0/1/2), aparameter estimator 889, subject models 887-0/1, a compare section 885,and an evaluation section 883. Data sources can include at least oneprimary, or direct data source (sensor 818) and one or more secondarydata sources (in the example shown, 808-1 and 808-2). The primary datasource 818 can provide data for a value that is inferred from secondarydata sources 808-1/808-2 and can serve as a reference value. Datasources (818, 808-1, 808-2) can provide both current data as well aspast data. In the embodiment shown, the primary data source 818 can be asensor. In some embodiments, the primary data source 818 can be a sensorthat makes a biophysical measurement of a subject.

Data source models (891-0/1/2) can include any suitable predictivestatistical learning agent that can predict future data values based onpast data values. Thus, data source models 891-0, 891-1, 891-2 generateinferred data values from past data values of their corresponding datasources 818, 808-1, 808-2, respectively.

Parameter estimator 889 can receive past data values from secondary datasources 808-1, 808-2, and be in communication with subject models 887-0and 887-1. Based on received data values, parameter estimator 889 canupdate parameters of the subject models 887-0 and 887-1. That is, theparameter estimator 889 can be used to train the subject models 887-0,887-1.

The subject model 887-0 can infer a predicted value of a biophysicalresponse based on predicted data values from data source models 891-1/2(i.e., predicted secondary data values). The subject model 887-1 caninfer a predicted value (e.g., a biophysical response such as a glucoseresponse) based on actual data values from data sources 808-1, 808-2(i.e., actual secondary data values). It is understood that the valuespredicted by subject models 887-0/1 can be for the same featuresmeasured by primary data source 818.

The compare section 885 can make comparisons with a reference valueprovided by primary data source 818. In the embodiment shown, comparesection 885 can include a number of compare operations 885-0 to -2.Compare operation 885-0 can compare reference values (from sensor 818)with predicted values from data source mode 881-0. Compare operation885-1 can compare the reference values with predicted value from firstsubject model 887-0. Compare operation 885-1 can compare referencevalues with a predicted value from second subject model 887-1.

Evaluation section 883 can receive the various comparisons from comparesection 885, and in response, update any of the models and/or parameterestimator 889 accordingly, with model adjustments 881.

FIG. 9 shows an evaluation system 900 according to another embodiment.The system 900 can be one particular implementation of that shown inFIG. 8. System 900 includes data sources 908-0 to 908-2′, data sourceLSTMs 991-0/1/2, parameter estimator 989, subject body models 987-0/1,GANs 979-0/1, signal compare section 985, block evaluator 983, andfeedback generator 977.

Data sources (908-0 to 908-2′) can include data for signals of threedifferent types, type X, type Y and type Z. The type X signal can be aresponse to be simulated by a system. Type Y and type Z signals can beused to infer predicted type X signals. In the embodiment shown, datasources (908-0 to 908-2′) can provide signals for different time periods(T0, T1) and (0, T0), with time period (T0, T1) following and, in someembodiments, overlapping with time period (0, T0). Data source 908-0′can be a first type sensor that provides a type X signal for a timeperiod T0, T1. Data source 908-0 can be sensor data for a type X signalfor a time period 0, T0. Data source 908-1′ can be a second type sensorthat provides a type Y signal for time period T0, T1. Data source 908-1can be sensor data for a type Y signal for time period 0, T0. Datasource 908-2′ can be data logged by a subject that provides a type Zsignal for a time period T0, T1. Data source 908-2 can logged data for atype Z signal for a time period 0, T0.

In some embodiments, data sources 908-0/0′ can provide data generated bya CGM of a subject, data sources 908-1/1′ can be data generated by anHRM of the subject, and data sources 908-2/2′ can logged food data fromthe subject.

LSTMs 991-0/1/2 can generate predicted type X, type Y and type Zsignals, respectively, for time period T0, T1, from actual signal datafrom a previous time period 0, T0.

Subject body models 987-0/1 can be ANN or other ML models of a subjectbiophysical response that generates a predicted type X signal from typeY and type Z signals. The first subject body model 987-0 can generate apredicted type X signal for a time period (T0, T1) from predicted type Ysignals 975-0 and predicted type Z signals 975-1 provided from LSTMs991-1 and -2, respectively. The second subject body model 987-1, whichcan be a copy of the first subject body model 987-0, can generate apredicted type X signal for time (T0, T1) from type Y and type Z signalsfor time period (T0, T1). Parameter estimator 989 can update parametersof subject body models 987-0/1 based on Type X, Y and Z signals for timeperiod (0, T0). In other words, subject body model 987-0 can predict thetype X signal based on predicted type Y and type Z signals, whilesubject model 987-1 can predict the type X signal based on actual type Yand type Z signals.

GANs 979-0/1 can take predicted type X signals provided by subject bodymodels 987-0/1 and adjust them to take a more realistic form. Forexample, GANs (979-0/1) can have been trained with actual type Xsignals.

A signal compare portion 985 can compare a reference type X signal 973from data source 908-0′, to various outputs provided from system 900 todetermine an accuracy of such blocks. As but a few examples, compareportion 985 can compare reference signal 973 to the predicted type Xsignal 971-0 from LSTM 991-0 to determine the accuracy of LSTM 991-0.Reference signal 973 can be compared to the predicted type X signal971-1 from subject body model 987-0 to determine the accuracy of thesubject body model 987-0 when operating with parameter estimator 989 andLSTMs 991-1/2 but without GAN 979-0. Reference signal 973 can becompared to the predicted type X signal 971-2 from GAN 979-0 todetermine the accuracy of the subject body model 987-0 with GAN 979-0.Reference signal 973 can be compared to the predicted type X signal971-3 from subject body model 987-1 to determine the accuracy of thesubject body model 987-1 without LSTMs 991-1/2 or a GAN 979-1. Referencesignal 973 can be compared to the predicted type X signal 971-4 from GAN979-1 to determine the accuracy of the subject body model 987-1 with GAN979-1, but without parameter LSTMs 991-1/2.

A signal compare portion 985 can perform various other compareoperations among the operational blocks of the system. For example, asignal compare portion could compare a type Y signal for time (T0, T1)from data source 908-1′ with a predicted type Y signal output from LSTM991-2 and/or a type Z signal for time (T0, T1) from data source 908-2with a predicted type Z signal output from LSTM 991-1.

A block evaluator 983 can determine if any system blocks (e.g., LSTMs991-0/1/2, parameter estimator 989, subject body models 987-0/1, or GANs979-0/1) is operating below a desired accuracy level from comparisonresults provide by signal compare portion 985. If a block is performingbelow a desired accuracy level, feedback generator 977 can generatefeedback signals 981 for the block. As but one example, an error measurefor a block can be back propagated through the blocks model with an aimof minimizing the error.

It is understood that a “signal” as described herein, can be a machinelearning signal, representing a time series expression of a measuredvalue, in a suitable format (e.g., vector, matrix, tensor, etc.).

Embodiments can also include systems and method for predictingbiophysical responses (i.e., a predicted observable) by observingbiophysical responses for a limited time to create a predictivephysiological model. FIG. 10 is a block diagram of a system and method1000 according to such an embodiment.

The system/method 1000 can be conceptualized as including a trainingportion 1069 and a prediction portion 1067. A training portion 1069 caninclude a biophysical model 1065 and parameter estimator 1089 and canreceive data for training from a subject 1030. In FIG. 10, data fortraining can include a first data source 1008-0, a second data source1008-1, and optionally, a third data source 1008-2. The first datasource 1008-0 can be a first type sensor and a second data source 1008-1can be a second type sensor. According to some embodiments, it isdesirable to use a first type sensor 1008-0 in a more limited fashionthan a second type sensor 1008-1. As but a few of many possibleexamples, the first type sensor 1008-0 can be more expensive, moredifficult to employ, or more difficult to access than a second typesensor 1008-1. For example, the first type sensor 1008-0 can be a CRM.In some embodiments, a first type sensor 1008-0 can provide moreaccurate data for a desired predicted observable than a second typesensor 1008-1.

Biophysical model 1065 can include an ANN, or any other suitablestatistical learning agent, with initially unknown parameters.Biophysical model 1065 can receive, as training data over a period oftime, data source 1008-1 (second sensor) and optionally, third datasource 1008-2. In response to such data, biophysical model 1065 cangenerate a simulated future observable 1057. Parameter estimator 1089can receive the simulated future observable 1057 as well as data fromthe first data source 1008-0 (first type sensor), which can reflect asubject's current state. Based on such inputs, parameter estimator 1089can generate error values which can be used to back propagate 1055through biophysical model 1065. Through such training, parameterestimator 1089 and biophysical model 1065 can arrive at time varyingparameters 1063 for generating a simulated future observable 1057 thatis personalized to the subject 1030.

A prediction portion 1067 can include a physiological model 1018 for asubject that uses the personalized time-varying parameters 1063developed by the training portion 1069. In some embodiments,physiological model 1018 can include an ANN, or any other suitablestatistical learning agent, having the same general structure asbiophysical model 1065. Physiological model 1018 can receive data fromsecond data source 1008-1 and optionally a third data source 1008-2. Inresponse to such data, physiological model 1018 can infer a predicatedobservable 1061. Thus, a predicted observable 1061 can be generatedwithout the use of a first data source 1008-0 (e.g., the morecostly/difficult to use first type sensor).

FIG. 11 is a block diagram of a system for predicting, in real-time, aglucose level of subject using personalized model, using only indirectdata sources. “Indirect” data sources are data sources that do notmeasure a glucose level directly. System 1100 can be one implementationof that shown in FIG. 10.

The 1100 can include a glucose-insulin regulatory model 1165 andparameter estimator 1189. Model 1165 can initially include unknownparameters (e.g., default parameters not particular to a subject orrandomized parameters). Model 1165 can receive training data from an HRM1108-1 and food data logged by subject 1108-2 to generate a simulatedglucose level 1157. Parameter estimator 1189 can utilize current CGMdata 1159 from a CGM sensor 1108-0 of a subject 1130 and the simulatedglucose level 1157 to back propagate 1155 through model 1165 to adjusttime-varying parameters of the model 1165. Once model 1165 generatessufficiently accurate simulated glucose level 1157, time-varyingparameters 1163 (which can be considered personalized to the subject1130 as the model 1165 is trained with subject data) can be provided toreal-time prediction portion 1167.

Real-time prediction portion 1167 can include physiological model 1118which can receive personalized time-varying parameters 1163 developed bytraining portion 1169. Using indirect data sources of HRM 1108-1 andfood logging 1108-2 (i.e., without the use of a direct measurement ofglucose levels from CGM sensor 1108-0), physiological model 1118 cangenerate a predicted glucose level 1161. In this way, glucose levels canbe predicted for a subject with less expensive, more accessible sensors.

While embodiments can include various systems and methods that utilizeML architectures for predicting and classifying subject behavior,embodiments can also include methods for providing data to such MLarchitectures.

FIG. 12A is a flow diagram of a method 1201 for processing data objectsaccording to an embodiment. The method 1201 can be performed by a systemof one or more appropriately-programmed computers in one or morelocations. The system can create one or more data processing objects inmemory (1201-0). Such an action can include transferring data stored ona nonvolatile storage unit, or accumulating data received from a user ina system memory of a ML computing system. The system can decorate thedata processing object to generate a message after it has been processedby an ML process (1201-2). Such an action can include transforming thedata processing object into a more complex object that includes amessaging function.

The system can determine if a ML process is available (1201-4). If aprocess is available (Y from 1201-4), the system can process thedecorated data processing object (1201-6). Such an action can includeapplying data to an ANN, or any other suitable statistical learningagent, for a training, inference, or other operation. The system canthen determine if a post-processing message for the data processingobject has been received (1201-8). Once a post-processing message hasbeen received (Y from 1201-8), the system can determine if a last dataprocessing object has been processed (1201-10). If there are more dataprocessing objects to be processed (N from 1201-10), the system canproceed to the next data processing object (1201-12) and can return to1201-2.

FIG. 12B is one example of code 1253 for decorating a data objectaccording to one embodiment. Such code can be executable by one or moreprocessors of a computing system. Code 1253 can include a function 1251,shown at (1) (“run service”), which can decorate a data processingobject to generate a message once it has been processed. At (2), thefunction can create an object in memory for receiving data to beprocessed. At (3), the data object in memory can receive data to beprocessed. At (4), a process target can be decorated to generate adesired message. At (5), one process of multiple forked processes can beinstantiated to operate on the decorated object. At (6), the process canbe started.

FIG. 12B also shows a function 1249-0 “service_encode” for encapsulatingan egress message generated by a decorated data processing object, aswell as the decorating function “post_process_decorator” 1249-1, whichcalls the message encoding function 1251-0.

FIGS. 12C and 12D are block diagrams of a system 1200 showing a dataprocessing operation according to an embodiment. Referring to FIG. 12C,system 1200 can include a system memory 1247, a machine learning service1241, and pre- and post-processing functions 1233. Data processingobjects can be created in system memory 1247. Data processing objectsshown as 1245′ have been processed to generate results 1235. Dataprocessing objects shown as 1245 have yet to be processed.

Before data within a data processing object is processed, by operationof a decorating function, a data processing object 1245 can betransformed into an object 1243 that includes a messaging function. Theobject 1243 can be processed by an available process 1239 of ML service1241. In some embodiments, processes of ML services 1241 can beasynchronous processes. Busy processes are shown as 1239′. A busyprocess that is ending (i.e., is the next process to be free) is shownas 1239″. Once processing of data processing object is complete, results(e.g., 1235) can be returned to system memory 1247. In addition, byoperation of the decoration, an egress message can be generated 1237.

Referring to FIG. 12D, the generation of egress message 1237 can be usedas an indication that a process is available 1239. A next dataprocessing object can be decorated 1243 and provided to the availableprocess 1239.

While embodiments above describe various methods, both explicitly andimplicitly, additional methods will be now be described with referenceto a flow diagram.

FIG. 13 is a flow diagram of a method 1301 according to an embodiment.The method 1301 can be performed by a system of one or moreappropriately-programmed computers in one or more locations, e.g., oneof the systems described previously in this disclosure. In an operation,the system can create a biophysical model for a subject with machinelearning by training with primary sensor data and secondary data sources(1301-0). This operation can include creating models according to any ofthe embodiments herein, or equivalents. In some embodiments, a primarysensor data source can provide data for a subject response that is thesame as that to be predicted. A secondary data source can provide datathat is not the same as that to be predicted. In some embodiments,primary sensor data can be more difficult to acquire than data fromsecondary data sources. Further, primary sensor data sources andsecondary data sources can be from a particular subject. Thus, thebiophysical model can be a model personalized to a particular subject.

The system can receive current data from a secondary data source(s)(1301-2). Such an action can include receiving sensor data from asubject. A subject response can then be predicted with the biophysicalmodel with at least the current secondary sensor data (1301-4). Thesystem can determine if a predicted subject response is outside of oneor more predetermined limits (1301-6). Such an action can includecomparing a predicted response to limits or goals established by asubject. Such goals can be personal goals, or goals dictated by healthneeds. If a predicted response is not outside of limits (N from 1301-6),the system can generate one type of predetermined output (1301-10). Inthe particular embodiment shown, this can include a message for thesubject indicating the subject is “on-track”. If a predicted response isoutside of limits (Y from 1301-6), the system can generate another typeof predetermined output (1301-8). In the particular embodiment shown,this can include a message for the subject indicating a possible actionto be taken. It is understood that outputs from actions 1301-08 and/or1301-10 can be to third-parties or intermediate parties (e.g., medicalprofessionals), as well as the subject.

In some embodiments, the system can predict the glucose level of asubject. A primary sensor can be a sensor that directly measures glucoselevels (e.g., CGM). Secondary sensors can be sensors that track subjectactivity (e.g., HRM and/or food logging) but not the response to bepredicted directly.

FIG. 14 is a flow diagram of a method of health management 1401according to an embodiment. The method 1401 can be performed by a systemof one or more appropriately-programmed computers in one or morelocations, e.g., one of the systems described previously in thisdisclosure. In an operation, the system can create one or more MLbiophysical models for a subject with direct data and indirect data(1401-0). In some embodiments, a direct data source can provide data fora biophysical response that is the same as one predicted in the method1401. An indirect data source can provide data that is not the same asthat to be predicted. In some embodiments, all or a portion of data froma direct data source and/or an indirect data source can be from aparticular subject to form a personalized biophysical model.

The system can set limits to a biophysical response based on thesubject's health (1401-2). Such limits can be static limits or dynamiclimits and can include rates of change. The system can receive currentindirect data for a subject (1401-4). The system can infer one or morefuture responses of the subject from the indirect data using the modelscreated in operation 1401-0(1401-6). The system can determine if apredicted response is outside of one or more response limits (1401-8).If a predicted response is not outside of limits (N from 1401-8), thesystem can return to 1401-4.

If a predicted response is outside of limits (Y from 1401-8), adeviation between the predicted response and the limits can bedetermined (1401-10). Based on such a quantified deviation, one or moreremedial actions can be determined 1401-12. A message can then be sentto the subject notifying the subject of the expected deviation alongwith one or more suggested remedial action (1401-14).

Optionally, if a predicted response is outside of limits (Y from H70-8),an iteration rate (e.g., a rate at which indirect data is received orsampled) can be increased (1401-16), and/or a third party can benotified (1401-18).

FIG. 15 is a flow diagram of a method of coaching a subject 1501according to an embodiment. The method 1501 can be performed by a systemof one or more appropriately-programmed computers in one or morelocations, e.g., one of the systems described previously in thisdisclosure. The system can create one or more ML biophysical models fora subject with first sensor data and subject data (1501-0). Such actionscan include creating models according to any of the embodiments herein,or equivalents. In some embodiments, first sensor data can be data froma sensor that takes biophysical readings directly from a human body.Subject data can be data provided by the subject.

The system receives goal-related limits from the subject (1501-2). Suchactions can include receiving health, behavior or other goals from asubject, and determining how such limits can be sensed with a predictedsubject response. Possible rewards for the subject can be received(1501-4). Such an action can include determining rewards based on asubject's personal preferences.

The system can also receive/infer possible actions for the subjectrelated to subject goals (1501-6). Such an action can includedetermining activities a subject prefers, but such actions can alsoinclude using and/or presenting for selection “canonical” actionsinferred as described herein, and equivalents.

Referring still to FIG. 15, subject data can be received (1501-8). Usingsuch received data, a subject future response can be inferred the MLmodel from 1501-10. The system can determine if a predicted subjectresponse it outside of one or more of the goal related limits (1501-12).If a predicted response is not outside of limits (N from 1501-12), thesystem can send a reward to the subject (1501-20). If a predictedresponse is outside of limits (Y from 1501-12), the system can send amessage to a subject encouraging actions to meet goals (1501-14). Inaddition, a message can be sent suggesting particular actions that canbe taken to meet goals (1501-16). Such particular actions can includeactions from 1501-6. In the particular embodiment shown, the system canoffer or indicate a reward for meeting goal(s) (1501-18).

Referring now to FIGS. 16A to 16C, a subject device application andmethod for data acquisition is shown in a series of diagrams. Referringto FIG. 16A, a user device 1610 can include an application 1628 storedthereon in memory, such as nonvolatile memory, for execution by one ormore processors of the user device 1610. While the user device 1610 isshown as a smartphone, the user device can take the form of any of thoseshown herein, or equivalents.

Referring to FIG. 16B, when an application is active, a user device canconnect to, or be connected to, one or more sensor devices 1620-0,1620-1. Sensor devices (1620-0, 1620-1) can sense biophysical responsesof a subject. In the particular example shown, the sensor devices caninclude an HRM and CGM. However, alternate embodiments can includesensors suitable for a desired modeled response. Data from sensordevices (1620-0, 1620-1) can be provided, directly or via one or moreother devices, to ML services 1602 for learning operations. Suchlearning operations can include any of those described herein orequivalents. In some embodiments, HRM and CGM data can be provided to MLservices to create a personalized glucose level response model which canpredict glucose levels.

Referring to FIG. 16C, an active application can also enable a subjectto log data related to a predicted biophysical response. Such loggeddata can be provided, directly or via one or more other devices, to MLservices 1602 for learning operations. In some embodiments, anapplication can provided various ways to log data values. In someembodiments, an application can enable food data (e.g., consumed food)to be logged by image capture 1622-0, voice entry 1622-1, or manually(e.g., enter text) 1622-2. However, such data entry methods should notbe construed as limiting. As noted herein, such ML models can infernutrition information from such logged data. That is, such data can alsobe used for initial inference operations which can yield nutrition datafor learning operations.

Referring now to FIGS. 17A to 17F, a subject device application forgenerating recommendations is shown in a series of diagrams. Referringto FIG. 17A, a user device 1710 can include an application 1728 storedthereon in memory, such as nonvolatile memory, for execution by one ormore processors of the user device 1710. While the user device 1710 isshown as a smartphone, the user device can take the form of any of thoseshown herein, or equivalents. Application 1728 can be the same as, ordifferent from, that shown in FIGS. 16A to 16C.

Referring to FIG. 17B, when an application is active, the user devicecan connect to, or be connected to, one or more sensor devices 1720-1.The sensor device 1720-1 can sense one or more biophysical responses ofa subject. In some embodiments, the sensor device 1720-1 can be anindirect data source, sensing a biophysical response different from abiophysical response utilized to predict subject actions and makerecommendations. In the particular example shown, sensor device 1720-1can be an HRM that can provide data for predicting glucose levels of asubject. Data from sensor device(s) 1720-1 can be provided, directly orvia one or more other devices, to ML services 1702 for inferenceoperations. Such inference operations can include any of those describedherein or equivalents. In some embodiments, HRM and other data can beprovided to ML services to predict glucose levels of a subject.

Referring to FIG. 17C, an active application can also enable the subjectto log data related to a predicted biophysical response. Data loggingcan occur in the same fashion as noted for FIG. 16C (e.g., image 1722-0,voice 1722-1, manual entry 1722-2). However, logged data can be providedto ML services 1702 for inference operations. In particular embodiments,logged food data and HRM data can be used to forecast a subject glucoselevel.

Referring to FIGS. 17D and 17E, in response to data from sensor 1720-1and/or logged data, an application 1728 can receive recommendations fromML services 1702. Recommendations can be derived from an inferenceoperation and/or from preferences or selections provided by a subject.In the embodiment shown, FIG. 17D shows an activity recommendation1731-0. FIG. 17E shows a nutrition recommendation 1731-1.

Referring to FIG. 17F, in the event a subject's predicted biophysicalresponse(s) is within a desired limit, an application can offer a reward1729. In some embodiments, a reward can be provided by an applicationserver 1704.

FIG. 18A is a block diagram of a personalized response model creationsystem 1840 according to an embodiment. The system 1840 can include anunpersonalized model section 1840-0, a personalized data section 1840-1,and a resulting personalized model 1854. The unpersonalized modelsection 1840-0 can include unpersonalized biometric data 1846, astarting biometric model 1848, a derived function 1850, and anunpersonalized biometric model 1852. The unpersonalized biometric data1846 can be data for a biophysical response over time for a generalpopulation, such as the rate at which one or more substances enter orare removed from the body or bloodstream. The starting biometric model1848 can be a model for predicting a biophysical response, and in someembodiments can be in the form of a differential equation. The startingbiometric model 1848 can include a number of functions, at least one ofwhich may be derived from unpersonalized biometric data 1846. In someembodiments, deriving the model can involve using a machine learningoperation (e.g., regression) to fits the unpersonalized data to afunction. A biometric model with the derived function can be created.Because such a model includes one or more functions based onunpersonalized biometric data, it can include unpersonalized parameters.

Personalized data section 1840-1 can include subject biometric sensordata 1842. Personalized biometric parameters can be extracted from thepersonalized data 1840-1. The biometric sensor data 1842 can be sensordata for a subject for which a personalized response will be predicted.The extracted personalized parameters 1844 can represent the sameparameters as the unpersonalized parameters of the model 1852.Extraction of personalized parameters 1844 can be accomplished withmachine learning that seeks to fit biometric sensor data 1842 to anexpected response. However, in other embodiments personalized parameters1844 can be determined by other means, such as a clinical test, as butone example.

The unpersonalized biometric parameters of 1852 can be substituted withthe extracted personalized parameters of 1844 to create a biometricmodel with the derived function and personalized parameter 1854.

FIG. 18A also shows a biometric response prediction system 1860. System1860 can utilize the model from 1854 to provide a personalized biometricresponse for the subject of 1840-1. Personal data for the derivedfunction 1856 can be provided to the model 1854, and the model cangenerate a personalized biometric response 1858. In some embodiments,system 1860 can execute an inference operation with the personal data1856 as input data.

While systems as described herein can be utilized to provide anysuitable personalized predicted biometric response, in some embodimentsa model can predict a glucose over time (glycemic) response. Such anembodiment is shown in FIG. 18B.

FIG. 18B shows a block diagram of a glycemic response model creationsystem 1840′ according to an embodiment. The system 1840′ can include anunpersonalized model section 1840-0′, a personalized data section1840-1′, and a personalized glycemic prediction model 1854′. Theunpersonalized model section 1840-0′ can include unpersonalized responsedata 1846′, a glucose regulation model 1848′, a derived food function1850′, and an unpersonalized glycemic prediction model 1852′. Theunpersonalized biometric data 1846′ can be unpersonalized glycemicresponses to food data 1846-0′ and/or other glycemic response data1846-1′, from the general population. Other glycemic response data1846-1′ can be data statistically calculated from glycemic index datafor food, as but one of many possible examples.

The glucose regulation model 1848′ can be in the form of a differentialequation that gives a glucose rate over time. As but one of manypossible examples, the glucose regulation model 1848′ can include aglucose production portion and glucose update portion. The food sourcefunction 1850′ can be derived from the unpersonalized response data1846′. In some embodiments, the food source function 1850′ can be afunction that expresses the generation of glucose in response toconsumed food. In some embodiments, such a function can be derived usinga machine learning operation (e.g., a regression model) that fits theunpersonalized data to a function.

One or more glucose regulation models 1852′ can be created using thederived function. Such a model can include the food source function1850′ as well as unpersonalized glycemic response parameters. In someembodiments, such parameters can be demographic equivalent parameters byderiving (e.g., training) the model with demographic equivalent datasets. However, in other embodiments, demographic equivalent parameterscan be “hidden” or embedded groupings that arise from unsupervised orsupervised training. In a particular embodiment, such parameters caninclude one or more insulin resistance parameters.

The personalized data section 1840-1′ can include personalized glucoseresponse data 1842′ and the extraction of personalized glycemic responseparameters 1844′. Personalized glucose response data 1842′ can includepersonal food stimulus data 1842-0′ and a corresponding personal glucosedata 1842-1′. In some embodiments, the personal food stimulus data1842-0′ can be data describing food eaten by a subject (e.g., foodlogging), while the personal glucose data 1842-1′ can be glucose levelsread by a sensing system, such as a continuous glucose monitoring (CGM)device, or some other glucose meter. Such data can represent a subject'spersonal glycemic response over time to the foods that the subject ate.The extracted personalized glycemic response parameters 1844′ cancorrespond to the unpersonalized glycemic response parameters of 1852′.In a particular embodiment, such parameters can include one or morepersonal insulin resistance parameters. The parameters can be derivedfrom the personal food stimulus data 22-0′ and the personal glucose data1842-1′.

The unpersonalized glycemic response parameters of 1852′ can besubstituted with the extracted personalized glycemic response parametersof 1844′ to create a personalized glucose regulation model 1854′. Insome embodiments, the personalized glucose regulation model 1854′ cantake the form of:

dG/dt=F _(food)(food,t . . . )+F _(produce)(G(t) . . . )+F_(uptake)(G(t) . . . )

where dG/dt is the rate of change of glucose levels (e.g., blood glucoselevels) in the body over time, F_(food) is a food source function thatcan depend on characteristics of food eaten (food) and time, F_(produce)can represent a body's glucose production, and F_(uptake) can representa body's glucose uptake. Any of the functions can include parameters asnoted herein. For example, the insulin resistance parameters can beincluded in a function F_(uptake).

FIG. 18B shows a resulting glycemic response prediction system 1860′.The system 1860′ can utilize the model 1854′ to provide a personalizedglycemic response for the subject that provided personalized data1842′t. Food source data 1856′ can be provided to the model 1854′, andthe model can generate a personalized glycemic response 1858′. In someembodiments, the system 1860′ can perform an inference operation withthe food source data 1856′ as input data in real time.

In this way, embodiments can generate predicted glucose levels that canbe more personalized than conventional approaches. In some conventionalapproaches, subject data can be extrapolated from a linear model, whichmay not capture the complexity of a glucose regulation system in themanner of a machine-learned solution, as described herein.

Further, embodiments herein present methods and systems that are easilyand readily adaptable to individuals. Once a glucose regulation modelhas been constructed from unpersonalized data, personalized parametersfor a subject can be incorporated into the glucose regulation model forpredicting glucose levels of the that subject. This is contrast toconventional approaches that may use training data generated by thesubject (e.g., food diary, blood glucose levels, activity) to create amodel for that same subject. Then the same data is required an inferenceoperation on the model. This is in sharp contrast to derivingpersonalized parameters and incorporating them into an existing model(constructed with unpersonalized data).

As noted herein, in some embodiments, personalized parameters for asubject can be demographic equivalent parameters. That is, features of asubject can be classified according to models created with large datasets to derive personalized parameters for the subject without having totest the subject. As but one of many examples, an insulin resistanceparameter could be derived by classifying an individual according to anyof various factors (e.g., age, sex, body size/type, lifestyle, location,place of family origin, know relatives, preferred diet, and numerousothers) to generate a demographic equivalent insulin parameter, withoutthe individual having to undergo a blood test, or the like.

While embodiments can include systems and methods for modeling andpredicting subject biometric responses, embodiments can also includesystems and methods for predicting a target feature of an item, withoutnecessarily having all attributes of the item.

FIG. 19A is a block diagram of a system and method for predicting afeature of an item according to an embodiment. FIG. 19A shows a trainingsection 1970 for creating a prediction model, and an inference section1972 for predicting target values. The training section 1970 can includetraining data 1922, a random selection of attributes 1962, an imputingAE 1964, and a bidirectional recurrent neural network (RNN) and/or LSTM1966 (hereinafter LSTM). The training data 1922 can include sets ofattributes and a corresponding target for multiple data items. That is,each data item may be composed of multiple attributes which give rise toa target (feature) for the item. The target can serve as a label fortraining operations. An attribute for an item can be randomly selectedfrom the training data. The AE 1964 can be trained to impute missingattributes from a subset of the attributes 1922, or from a corrupted setof the attributes 1922. The training process can involve providing thesubset of attributes or the corrupted attributes to an untrained model,mapping such attributes to a hidden representation, and attempting toreconstruct the complete or uncorrupted attributes from the hiddenrepresentation. The reconstruction can be compared to the actualattributes, and the parameters of the model can be updated accordingly.This process can be repeated until convergence, i.e., untilreconstruction error satisfies a criterion. In some cases, trainingattributes can be selected randomly (1962). The bidirectional LSTM 1966can be trained on imputed attributes from AE 1964 along with a currentlyselected attribute to predict the target for the item from whichattributes are selected.

An inference section 1972 can include a trained AE 1964 and trainedbidirectional LSTM 1966. A subject can input attributes of an item 1968to imputing AE 1964 and bidirectional LSTM 1966. As each attribute isinput, imputing AE 1964 can impute missing attributes. In response toeach input attribute and the imputed attributes, the bidirectional LSTM1966 can generate a target feature 1969. In this way, the bidirectionalLSTM 1966 can predict a target from an incomplete set of attributes.

While systems as described herein can be utilized to predict a targetfeature for any suitable items having multiple attributes, in someembodiments a system and method can predict a glycemic value for a fooditem. Such an embodiment is shown in FIGS. C2A and C2B.

FIG. 19B shows a training section 1970′ for creating a glycemic responsemodel. Training section 1970′ can include training nutrition data 1922′,a random nutrient selection 1962′, a denoising imputing AE 1964′, and abidirectional LSTM 1966′. Training nutrition data 1922′ can includenutrients and a corresponding glycemic value for various food items. Theglycemic value can be any suitable value related to a glucose responseof a person (e.g., blood glucose), and in some embodiments can include aglycemic index (GI), glycemic load (GL), or both. The denoising andimputing AE 1964′ can be trained to impute nutrient values for an itembased on randomly selected nutrients. The bidirectional LSTM 1966′ canbe trained with imputed nutrients from denoising imputing AE 1964′ and acurrently selected nutrient to predict a glycemic value.

Referring to FIG. 19C, a glycemic value prediction method and system1972′ can include a denoising imputing AE 1964′ and bidirectional LSTM1966′ trained as noted above in FIG. 19B. In an operation, a subject caninput nutrients of a food item 1968′ to denoising imputing AE 1964′ andbidirectional LSTM 1966′. As each nutrient is input, denoising imputingAE 1964′ can impute additional nutrients. In response to each inputnutrient and the imputed nutrient, bidirectional LSTM 1966′ can generatea predicted glycemic value. In some embodiments, glycemic values (GI/GL)can be provided to a subject in real time as each nutrient is entered.

While embodiments can include systems that can impute attributes topredict a target feature of an item, embodiments can also includesystems and methods for imputing missing sensor data for a subject usingmultiple sensors.

FIG. 20A is a block diagram of a system and method 2076 for imputingmissing sensor data according to an embodiment. The system 2076 caninclude one or more sensors 2078, ML embedding system 2080, MLimputation system 2082, and normalization system 2084. Sensor(s) 2078can include one or more sensors that detect a biophysical response of aperson over time. In some cases, the biophysical response data may bemissing, corrupt, or otherwise determined to be not available or valid.ML embedding system 2080 can embed data from the sensors 2078 using anANN. In this disclosure, the word “embed” or “embedding” may refer to aprocess by which data is mapped to vectors of real numbers. Theresulting vector space may have a lower dimension than the input. Assuch, embedding may be considered a dimensionality reduction technique.When there are multiple sensors 2078, different sensor values can begrouped together (e.g., concatenated) during the embedding operation.

The ML imputation system 2082 can receive embedded values from the MLembedding system 2080 and impute values for any missing sensor readings.In some embodiments, the ML imputation system 2082 can include an AEsimilar to the AE of FIG. 19. The output of ML imputation system 2082can be normalized with normalization system 2084. The resulting outputcan be imputed data 2086 which can include values that were not presentin sensor data provided to the system 2076.

While systems as described herein can be utilized to impute data valuesfor any suitable type of sensor data, in some embodiments the system andmethod can impute data for glucose and/or heart rate monitoring.

FIG. 20B is a block diagram of a system and method 2076′ for imputingmissing sensor data according to an embodiment. The system 2076′ canreceive data from multiple different sensor types 2016, 2018. In oneembodiment, sensor 2016 can be a CGM and sensor 2018 can be an HRM. TheML embedding system 2080′ can concatenate data from sensors 2016/2018and embed them into single values using a neural network. In someembodiments, ML imputation system 2082′ can include a stacked denoisingAE. The normalization system 2084′ can normalize the output of MLimputation system 2082′ to generate imputed data 2086′.

FIG. 20C shows sensor data 2018 and 2016 prior to processing by a system2076′. As shown, sensor data 2016 has a missing portion 417. FIG. 20Dshows sensor data 2016′, which can include imputed data 419 that hasbeen provided by operation of the system 2076′.

While embodiments can include systems that can impute missing datavalues from a sensor data set, embodiments can also include systems andmethods for determining a quality of a data set. FIG. 21A is a blockdiagram of a method and system 2188 according to such an embodiment.

The system 2188 can include a database system 2190 and an electronicdata processing system 2192. The database system 2188 can include one ormore good data sets 2194 (p) and a query data set 2193 (q). The gooddata set 2194 can be a high-quality data set. The query data set 2193can be a data set for evaluation. In some embodiments, data sets2193/2194 can be labeled data sets.

Electronic data processing system 2192 can include a classifier section2196 that gives a quality score 2198. The classifier section 2196 caninclude a neural network configured as a classifier. The classifier canbe conditioned on both data values and corresponding labels for the datavalues. The distribution for the classifier can be p(X, Y, Z) where Xcan represent the input feature distribution, Y can be a categoricaltarget, and Z can vary according to the data set. In some embodiments, Zcan be a binary variance with Z=1 if a sample (x,y) is from a query dataset (q), and Z=0 if a sample is from a validated data set (p). Thus, inthe binary case, a classifier can be built to give h(x)=p(z=1|x, Y=1).This is in contrast to a conventional classifier that can assumedistributions from good and query data sets are the same (i.e.,p(X,Y|Z=0)=p(X,Y|Z=1)) and is built for h(x)′=p(y=1|x). The qualityscore 2198 can be a quality value determined by classifier section 2196.For example, in the above binary case, if h(x)=0.5 for all x in eitherthe query dataset (q) or good dataset (g), the distributions can bedetermined to be indistinguishable, thus the query data set (q) can beconsidered high quality.

A software agent 2199 can then accept or reject the query data set (q)based on a generated quality score. Such an action can further includecopying the query data set to a database for use in training, inferenceor other operations.

While systems as described herein can be utilized to determine a qualityof various types of data sets, in some embodiments a system and methodcan determine a quality of data sets that include biophysical sensordata.

FIG. 21B is a block diagram of a system and method 2188′ for determininga quality of glucose level data with corresponding food log labels. Thesystem 2188′ can include a database system 2190′ and an electronic dataprocessing system 2192′. The database system 2190′ can receive datavalues from sensors 2116 and logged data 2120 and include a good dataset 2194′ (p) and a query data set 2193′ (q). In some embodiments, datasets 2194′/2193′ can be CGM data corresponding to food log data.

Electronic data processing system 2192′ can include a linear classifier2196′ that generates a quality score 2198′. Linear classifier 2196′ caninclude a neural network configured as a classifier similar to thatdescribed in FIG. 21A. Quality score 2198′ can indicate howdistinguishable the data sets were according to linear classifier 2196′.The quality score 2198′ can vary according to types of data sets, andcan take the form of those described herein, or equivalents. An agent2199′ can determine whether query data set 2198′ is accepted orrejected.

Embodiments can further include methods and systems that can predict asubject's behavior based on sensor signals. FIG. 22A is a block diagramof a method and system 2201 according to such an embodiment.

The method/system 2201 can include sensors 2216, subject logging data2220 and a prediction system 2205. Sensors 2216 can include one or moresensors that record a biophysical response of a subject 2203. Subjectlogging data 2220 can record behaviors of a subject 2203.

The system 2205 can train a classifier 2207 to predict a behavior 2213from a biophysical response 2216. The training data can be previousbiophysical responses 2216 labeled with resulting previous behaviors2220.

While systems as described herein can be utilized to determine any ofvarious behaviors in response to sensor data, in some embodiments asystem and method can predict food logging data in response to glucoseand heart rate data from a subject.

FIG. 22B is a block diagram of a system and method 2201′ that caninclude a prediction system 2205′ that receives sensor data and foodlogging data 2220′ from a subject 2203′. The sensor data can be from aglucose meter 2216′ and HRM 2218.

Prediction system 2205′ can train a classifier 2207′ to derive signatureglucose and heart rate signals for corresponding food ingestion periods2209′. Predictor system 2207′ can receive glucose sensor data and HRMdata for a same time period, and in response predict an ingested food2213′ related to the time period.

Embodiments can further include systems and methods having machinelearning models for determining the composition of items from a textdescription of the items. FIG. 23A is a block diagram of a system 2315according to such an embodiment.

The system 2315 can include a data input section 2317, a processingsection 719, and a formula database 2321. The data input section 2317can acquire text-related data regarding an object. In some embodiments,the data input section 2317 can include voice data 2323, image data2325, or text data 2327 from a subject. In some embodiments, such datacan be acquired by a subject device (e.g., smartphone).

The processing section 2319 can transform non-text data into text data.Such processing can include voice processing 2329 to derive text datafrom audio input data or optical character recognition 2331 to derivetext data from image data. While such processing can be performed by aremote system, all or a portion of such processing can also be performedby a subject device.

The processing section 719 can also include a machine learning naturallanguage processing (NL) parser 2333 and a query engine 2335. NLP parser2333 can determine a structure of input text. In some embodiments, theNLP parser 2333 can parse the text to determine its constituent wordsand can arrange such words in a particular order or format in response.The query engine 2335 can provide the arranged words to a formuladatabase 2321 to determine an object corresponding to the text. In someembodiments, the query engine 2335 can generate a list 2339 of possibleobjects (e.g., prioritized list).

While systems as described herein can be utilized to determine thecomposition of items based on text descriptions of the items, in someembodiments the composition of food can be determined from a textdescription of the food, such as a menu description. FIG. 23B shows anexample of a method/system 2315′ according to such an embodiment.

The method/system 2315′ can receive a text string description of a food2327. In some embodiments, such a text string can be a menu itemdescription. A machine learning NLP system 2333′ can parse the textstring. Such parsing can include determining and prioritizing nominativewords 2343-0 and non-nominative words 2343-1. Such processing can alsoinclude determining title nominatives, ingredient nominatives, andcertainties with respect to such words. Based on such parsing, the NLPsystem 2333′ can determine the presence and certainty of nominativewords, and if so, prioritize such words, including title nominativeswith certainties 2345-0 and explicit ingredients with certainties2345-1.

Query engine 2335′ can execute a sequence of query operations to arecipe database 2321′ using parsed text data. In the embodiment shown,query operations can be prioritized, starting with title nominativeswith certainties 2335-0. Query operations can then include secondaryqueries based on non-nominative prioritized words 2335-1. Query resultscan be filtered using explicit ingredients with certainties. In responseto each query, a list 2339′ of corresponding recipes can beprogressively refined’.

The system can select a recipe 2347 from the list 2339′. In someembodiments, this can include selecting a recipe having a best matchfrom the list. However, in other embodiments such an action can includea subject confirming or selecting a recipe from the list. A selectedrecipe can be applied to a nutrition inference system 2349 which cangenerate nutrition information (e.g., GI, GL) for the selected recipe.

Embodiments can further include systems and methods for determining theproportion of constituents in an item, based on features of such items.FIG. 24A is a block diagram of a method and system 2451 according tosuch an embodiment.

The method/system 2451 can receive data for an item composed of multipleconstituents. Such data can include cost and/or reward values for theoverall item 2453 and ranked constituent data 2455. Within an inferencesection 2457, given data 2453/2455, for each constituent of the item,the various cost/rewards for the item can be looked up to create amatrix 2457-0. In the embodiment shown, the item can include mconstituents, and there can be n cost/rewards for the item, thus alookup operation can generate an n×m matrix A. Using the generatedmatrix, the method/system 2451 can solve a system of equations 2457-1for each cost/reward (e.g., y=Ax), with the constraints imposed by theknown rank of ingredients (e.g., x1>x2 . . . >xm). Such an action caninclude instructions executed by a computing machine that solve thesystems of equations according to any suitable technique. In someembodiments, a neural network can be used to derive equationcoefficients using machine learning. Solving the system of equations canyield the amount of each constituent in the item 2459.

While system and methods as described herein can be utilized todetermine the amounts of ranked constituents of any item given suitablecost/reward data, in some embodiments, the composition of a food itemcan be determined based on nutrition data for such a food item and alist of ingredients in the food item, such as that present in a foodlabel. One such system is shown in FIG. 24B.

The system/method 2451′ can include a data acquisition section 2461 andprocessing section 2457′. The data acquisition section 2461 can includean image capture device 2461-0 and image processing section 2461-1. Theimage capture device 2461-0 can be used to capture an image of a fooditem label. The image processing section 2461-1 can derive food itemdata from a captured label, including n food item nutrition facts (y1,y2 . . . yn) 2453′ and m food item ranked ingredients (x1, x2 . . . xm)2455′.

Inference section 2457′ can lookup nutrient information for eachingredient to create an n×m matrix A for all ingredients 2457-0′. As inthe case of FIG. 24A, a resulting system of equations for eachingredient (e.g., y=Ax) can be solved, with the rank of ingredients asconstraints (e.g., x1>x2 . . . >xm). The resulting solved equations cangive the amount of each ingredient in the food item (i.e., its recipe).A derived recipe can be provided to a nutrition inference system 2449which can generate nutrition information (e.g., GI, GL) for the selectedrecipe.

FIG. 24C shows an example of food item data that can be processedaccording to embodiments. A food item 2463 can include a rankedingredient list 2455″ as well as nutrition facts 2453″. However, theamount of each ingredient in the ingredient list 2455″ is not known.Such data can be captured and processed by a method/system 2451′ toinfer the amount of each ingredient.

Embodiments can further include systems and methods for creatingnutritionally sensitive word embeddings for processing word stringsrelated to food.

FIG. 25A is a block diagram a method and system for training operations2565 according to an embodiment. FIG. 25B is a block diagram of a methodand system for an inference operation 2567 according to an embodiment.Referring to FIG. 25A, the training method/system 2565 can include fooddata input 2569, a word embedding system 2571, and resultingnutritionally sensitive food string embedding 2573. The word embeddingsystem 2571 can include an embedding section 2571-0 and weighing matrix2571-1. The food data input 2569 can include word strings and nutritionfacts for foods

Within embedding system 2571, embedding section 2571-0 can embed foodstring values according to any suitable technique, including word2vec,as but one example. The weighing matrix 2571-1 can be included intraining operations so that the nutrition facts corresponding to foodstrings are weighted in the word embedded space. Once trained, embeddingsystem 2571 can provide nutritionally sensitive food string embedding2573.

Referring to FIG. 25B, in an inference operation a query food string2577 can be applied to a trained embedding system 2571, to generate aword embedding that is nutritionally sensitive 983.

While embodiments above describe various methods, both explicitly andimplicitly, additional methods will be now be described with referenceto a flow diagram.

FIG. 26 is a flow diagram of a method for completing data sets 2685according to an embodiment. The method can be performed by a system ofone or more computers in one or more locations. The system can receivebiometric or food data sets (2685-0). Such an action can includereceiving any of the data set types described herein, including but notlimited to: CGM data sets, HRM data sets, and/or food logs. The systemcan evaluate such data sets (2685-2). Such an action can includedetermining if there are gaps in data sets, or data sets otherwiseexhibit low quality. In some embodiments, this can include qualitydetermination methods as described herein, or equivalents.

If a data set is determined not to be complete (N from 2685-2), valuescan be inferred and/or imputed to form a complete data set 2685-4.Complete data sets (Y from 2685-2 or 2685-4) can then be used to predicta biometric response 2685-6. In some embodiments, a biometric responsecan include blood glucose levels of a subject, including a personalizedglucose response as described herein, or an equivalent.

FIG. 27 is a flow diagram of a method for deriving nutrition data forfood items 2787. The method can be performed by a system of one or morecomputers in one or more locations. The system can receive food data(2787-0). Such an action can include a subject entering or otherwiseacquiring data related to a food item according to any of theembodiments described herein or equivalents. The system can determine ifnutrition data for the food item is in a database (or otherwise alreadyknown or available) (2787-2). If nutrition data is not known (N from2787-2), the system can generate or infer nutrition value for the fooditem using the food data 2787-4. Nutrition data (Y from 2787-2 or2787-4) can then be used to predict a biometric response 2787-6. In someembodiments, a biometric response can include blood glucose levels of asubject, including a personalized glucose response as described herein,or an equivalent.

FIG. 28 is a block diagram of a system for creating training data for aninference system for predicting a target value “Y” from mismatched data,as well as the inference system itself. The mismatched data can includefirst data that defines one or more attributes (e.g., text-baseddescriptions) of items (e.g., food items) and second data that definestarget values to be predicted (e.g., glycemic values) for such items orsimilar items. In some embodiments, the attributes are easier to obtainthan the target values.

The system 2800 can include an input data section 2806, embeddingsection 2818, training data 2806-2, and learning agent (e.g., NN) 2820A.Input data section 2806 can include mismatched data sets, including afirst data set 2806-0 that includes items with attributes “X” and seconddata set 2806-1 that includes items of the same type with targets “Y”.In this disclosure, the term “mismatched” may mean that attributes X fora particular item are known, but a target Y for the particular item isnot known. Alternatively or additionally, the term “mismatched” may meanthat the target Y for a particular item is known, but attributes X forthe particular item are not known. In some cases, both X and Y may beknown for a particular or for items with similar descriptions. Items ofdata sets 2806-0 and 2806-1 can be different and can have differentidentifying values. In some embodiments, data sets 2806-0/1 can includetext values or text values that have been encoded as numerical values.

Embedding section 2818 can match items from data sets 2806-0 and 2806-1by embedding the identifying values. In some embodiments, such actioncan include utilizing a neural network to generate embedded values thatwill correspond to attributes and targets. Data generated with embeddingsection 2818 can be stored as training data 2806-2. Training data 2806-2can be used by a training agent 2832 to conduct supervised training on aneural network 2820A to predict target values Y from attribute values X.A trained neural network 2820B can then be used to predict targets Yfrom attributes X, without having to identify an item, but rather enterattributes of the item.

In some embodiments, the system 23300 can predict a glycemic value fromnutrient data as shown in more detail in FIG. 29.

The system 2800 can include an input data section 2906, an embeddingsection 2918, training data 2906, and a learning agent 2932. The inputdata section 2906 can include at least one data set 2906-0 comprisingdescriptions of food items with nutrition information and at leastanother data set 2906-1 comprising glycemic data for such food items,similar food items, or different food items. Glycemic data can include aglycemic index (GI) and/or a glycemic load (GL). The data sets 2906-0and 2906-1 can include the same or different food items.

The embedding section 2918 can map the word descriptions of food itemswith nutrition information and the word descriptions of food items withglycemic values to vectors of real numbers in a high-dimensional vectorspace. The embedding section 2918 can do so by using an unsupervisedlearning algorithm (e.g., a clustering or dimensionality reductionalgorithm) or a neural network model, for example. Examples of suchunsupervised learning algorithms are bag-of-words models and n-grammodels. Examples of such neural networks are deep neural networks,autoencoders, or the like.

The distance between two vectors may represent the similarity of thedescriptions of the food items represented by the two vectors. Such adistance can be used to infer the glycemic value of a food item forwhich the glycemic value is not otherwise known, or the nutritioninformation of a food item for which the nutrition information is nototherwise known. In some cases, both the nutrition information andglycemic value for a particular food item may be known and need not beinferred. The vectors of real numbers, which may be referred to asembeddings in this disclosure, can be used as training data 2906-2 bythe supervised learning agent 2932. The labels for such embeddings canbe the inferred or known glycemic values. The learning agent 2932 cantrain a neural network 2920A to infer GI or GL values from nutrientdata. The resulting trained neural network 2920B can have nutrient factsas inputs and infer glycemic values.

FIG. 30 is a block diagram of a model creation system and method 3000according to an embodiment. The system/method 3000 can include aninitial model creation section 3036 and a model modifying section 3038,which can create a model 3042 for predicting data values. Thesystem/method 3000 can create a model based on four different timeseries data sets: A, B, C and D. Data sets A & B can be related totraining data sets 3034-0, and datasets C & D can be related trainingdata sets 3034-1. In some embodiments, any or all such data sets can betime series data generated by one or more biophysical sensors. In someembodiments, data for data sets A and B alone may not be sufficient tocreate a satisfactory predictive model. As but one of many possibleexamples, either or both of data sets A and B can have gaps where datais incomplete or otherwise erroneous.

The initial model creation section 3036 can create an initial model (G0)with data sets A and B. In particular, using M sets of training data, amodel can be trained to predict values B given values A. In someembodiments, such actions can include supervised training of a neuralnetwork model.

The model modifying section 3038 can use an initial model to createanother model using different data sets. In the embodiment shown, thiscan include using the initial model G0 as a baseline and retraining orcontinuing to train the model G0 with data sets C and D to create a newmodel 3038-0. This can include using N−1 data sets of C and D to trainthe new model (G) to arrive at or approach values D given values C. Insome embodiments, such actions can include supervised training of aneural network model. The model modifying section 3038 can further testthe new model G data values from different training sets 3038-1. In theembodiment shown, such an action can include testing the model on N+Mdatasets of A and C. Such testing can include iterating through toconvergence with sections 3038-0 and 3038-1. Through iteration andconvergence a best model can be arrived at 3042 (e.g., lowest errormodel).

Time series input data set A′ 3006 can be applied to best model 3042 toarrive at a predicted data set B (3044). In some embodiments, input dataset A′ 3006 can be sensor data of a subject and predicted data set B canbe a predicted biophysical response for the subject.

While systems as described herein can be utilized to create models usingdifferent data set types, in some embodiments a model can predict timeseries data for a sensor of one modality type, using training data fromsensors of different modality types. Such an embodiment is shown in FIG.31.

FIG. 31 shows a block diagram of a model creation system and method 3100according to an embodiment. The system/method 3100 can predict timeseries sensor data of a modality B from time series sensor data ofmodality A. Time series sensor data of different modalities can besensor data acquired with different sensor types and/or sensor dataacquired using different procedures.

The method/system 3100 can include an initial ML model section 3136 andan ML model modifying section 3138. Such sections can use training timeseries sensor data sets of different modalities A, B, C and D to createa model to predict sensor data sets of modality B from sensor data setsof modality A. Initial ML model section 3136 can receive training timeseries sensor data of modality B 3134-0 and modality A 3134-1. The model(G0) can be trained on M sets of data to predict modality B frommodality A. In some embodiments, training data sets A and B are notsufficient to ensure the creation of a model of sufficient accuracy fora desired result. A resulting model (G0) can be used as a baseline model3136-1.

Within the ML model modifying section 3138, an inverse of model G0 canbe used, referred to as inverse model “/G”, to estimate time series dataanalogous to modality A from different modality data 3138-0. In theembodiment shown, a mixture of time series data of modalities C and D3134-2, can be used on the inverse model /G. Using analogous modality Adata generated with the inverse model /G 3138-1, and time series data ofmodality C 3134-3, the inverse model /G can be trained to estimatelinear fitting parameters 3138-2 to generate analogous modality A datafrom modality C data 3134-3. Using the estimated linear fit parameters,analogous modality A data can be generated that is mapped to modality Cdata 3138-5.

A model G (of which /G is the inverse) can then be trained using timeseries of mixed modalities C & D 3134-2 and the analogous modality Adata 3138-5. In particular, the model G can be trained to generate timeseries of mixed modalities C & D from the analogous modality A data from3138-4.

An error can be calculated for model G on M+N data sets, and based onsuch error, the model G can be updated 3138-6. Based on the updatedmodel, a method/system can return to 3138-0, to generate a revisedinverse model /G. Such a process can continue to iterate 3146 until amodel of minimum error is generated 3142.

Time series sensor data of modality A 3106 can be applied to the minimumerror model G 3142 to generate time series sensor data of modality B3144.

In some embodiments, two or more time series sensor data can include adifferent types of glucose meters generating glucose levels.

While embodiments can include systems and methods for modeling andpredicting time series data, embodiments can also include systems andmethods for correcting time series data sets that can be subject toerror over time.

FIG. 32 is a block diagram of a system 3200 for correcting time seriesdata according to an embodiment. The system 3200 can include a modelsection 3252 that can create a correcting model by training with correct(or corrected) time series data 3248-0 and raw time series data 3248-1,which may have inherent or introduced error.

The model section 3252 can train a calibration model 3252-1 to generatecorrected or calibrated time-series data 3254 from time-series sensorraw data 3208-0. The model can be trained on training sets of raw(3248-1) and corrected (3248-0) time series data. The training sets ofraw time series data 3248-1 can be provided as inputs to the untrainedor partially trained calibration model 3252-1, and the training sets ofcorrected time series data 3248-0 can serve as labels. The outputproduced by the untrained or partially trained calibration model 3252-1can be compared to the labels, and based on the difference, which may bereferred to as an “error” or “loss,” the parameters of the calibrationmodel 3252-1 can be updated. This process can be repeated until theerror or loss is consistently small. In some cases, the calibrationmodel 3252-1 can be a deep learning neural network.

Trained calibration model 3252-1 can then be deployed to calibrate timeseries data to compensate for error. In particular, time series sensorraw data 3208-0 can be applied to the model to generate calibratedtime-series data 3254. In some embodiments, such an action can includeapplying time series sensor raw data 3208-0 as input data in aninference operation on a neural network-based calibration model.

While systems and methods as described herein can be utilized to correctany suitable set of time series data, in some embodiments a system andmethod can correct for drift in glucose level data generated by aglucose meter. Such an embodiment is shown in FIGS. 8A to 8C.

FIG. 33A is a block diagram of a system 3300 for calibrating raw timeseries glucose data according to an embodiment. The system 3300 caninclude a model section 3352 for creating a model by training with setsof corrected time series glucose data 3348-0 and raw time series glucosedata 3348-1. In some embodiments, raw time series glucose data 3348-1can be generated by a glucose meter that can drift over time. Such driftcan arise from a hysteretic effect in the glucose meter that canintroduce a dynamic error into sensor readings. Corrected time seriesglucose data can be data that has be validated, and so is known to beaccurate.

The model section 3352 can train a statistical ML model with thetraining sets of corrected time series glucose data 3348-0 and raw timeseries glucose data 3348-1 to infer time-variant drift cancellationparameters 3352-0. Optionally, the model section 3352 can includedomain-specific engineering 3356. However, other embodiments can includeautomatic feature extraction. Parameters derived at 3352-0 can be usedto create a drift cancellation model 3352-1.

Drift cancellation model 3352-1 can then be deployed to calibrate timeseries glucose data 3354 from raw time series glucose data 3308-0. Inparticular embodiments, raw time series glucose data 3308-0 can beapplied as input data in an inference operation to drift-cancellationmodel 3352-1.

FIG. 33B shows raw time series glucose data 3308-0 which can be appliedto a trained drift cancellation model 3352-1. FIG. 33C showscorresponding calibrated time series glucose data 3354 generated byoperation of a trained drift cancellation model 3352-1 on the raw timeseries glucose data 3308-0.

While embodiments can include systems and methods that can calibratetime series data to account for inherent error, embodiments can alsoinclude systems and methods for organizing data sets to enable theidentification/searching of such data sets for concerted events.

FIG. 34 is a block diagram of a system and method 3400 for organizingdata sets according to an embodiment. The system 3400 can include anoperational section 3458 and a data set source 3460 and can extractevents across multiple data sets. Data set source 3460 can include adata storage system configured to store a number of data sets withordered indexing. In some embodiments, data sets can represent eventsoccurring according to the ordered index. In some embodiments, data setscan be tabular data sets.

The operational section 3458 can access the data sets to build a datastructure with an interval tree-like structure and metadata 3458-0. Insome embodiments, such an action can include basing the tree-likestructure on the ordered indexing to enable rapid access to, andevaluation of, data values corresponding to the indexed locations.Metadata can provide information for the particular data set and/orrelate the particular data set to other data sets. The operationalsection 3458 can also find any missing intervals for the data structuresand impute data for the missing intervals 3458-1 to form fully populateddata structures (with respect to the ordering).

With the formation of interval tree-like data structures, the operationsection 3458 can execute operations between multiple data structures3458-2. In some embodiments, such operations can include, but are notlimited to: selecting between data structures, searching particularintervals over multiple data structures, combining portions of datastructures, or merging portions of data structures. From a datastructure having imputed values (3458-1) or operation results (3458-2),the data structure can be transformed into a tabular data format 3458-3.Such a tabular data format 3458-3 can be a representation of eventsacross data sets 3462 in indexed order.

While systems as described herein can be utilized to organize anysuitable data sets, in some embodiments, a system and method canorganize time series data to enable the generation of a tabular data setrepresenting concerted events over a selected time interval.

FIG. 35 is a block diagram of a system and method 3500 for representingdata sets in tabular form according to another embodiment. The system3500 can access a storage system that stores tabular data sets with atime and/or date column 3560. In some embodiments, one or more such datasets can be a biophysical sensor reading of a subject.

The operational section 3558 can organize data values of data sets 3560to enable rapid searching and access to the data values of the datasets. The operational section 3558 can create a data structure includingan interval tree using the time column and a desired sample rate 3558-0.In some embodiments, such a data structure can be contained as adataframe. Data encapsulation and inheritance from the original data setcan be maintained (e.g., with metadata of the data set).

Once data structures are created, in the event any data structures donot include time/date points of a desired range, an operational section3558 can automatically create such time points. In some embodiments,missing time points can be based on a sampling rate, however any othersuitable criteria can be used (e.g., force all data sets to have thesame or equivalent time/date points). Two class members can be created:INVALID, for those data structures missing time/date points, and VALID,for those data structures having all desired time/date points. ForINVALID data structures, data values can be imputed for the missingtime/date points 3558-4.

An operational section 3558 can enable any of various accessible memberfunctions 3558-2, including but not limited to: query another interval;query a time range; union like operations; and merging overlappingintervals.

From created data structures, an operational section 3558 can create adataframe 3558-3. From dataframes, concerted events across datasets canbe represented in a tabular format with a time interval column 3562.

FIGS. 36A to 36E show data operations according to an embodiment. It isunderstood that the data operations are provided by way of example andshould not be construed as limiting.

FIG. 36A shows a VALID data structure 3664. The data structure 3664 caninclude an interval tree corresponding to 24 sampling time periods. Datastructure 3664 can be created from a tabular data set having a datavalue DATA1 corresponding to each time period. FIG. 36B shows an INVALIDdata structure 3666. The data structure 3666 can be missing data forsampling time periods. Data structure 3666 can be created from a tabulardata set having a data value DATA2 for some, but not all time periods.As a result, there are missing time periods (shown by dashed lines).

FIG. 36C shows an integrated data structure 3666′ created from datastructure 3666 shown in FIG. 36B. Missing DATA2 intervals have beencreated. In addition, data values for missing intervals have beenimputed. FIG. 36D shows another integrated data structure 3668 createdfor another data set composed of data values DATA 3.

FIG. 36E shows a representation across data sets represented in tabularformat for times 12-15, generated from data structures of FIGS. 36A, 36Cand 36D.

Computer Systems

The present disclosure provides computer systems that are programmed toimplement methods of the disclosure. FIG. 37 shows a computer system3701 that is programmed or otherwise configured to implement the machinelearning models and methods described herein. The computer system 3701can be an electronic device of a user or a computer system that isremotely located with respect to the electronic device. The electronicdevice can be a mobile electronic device.

The computer system 3701 includes a central processing unit (CPU, also“processor” and “computer processor” herein) 3705, which can be a singlecore or multi core processor, or a plurality of processors for parallelprocessing. The computer system 3701 also includes memory or memorylocation 3710 (e.g., random-access memory, read-only memory, flashmemory), electronic storage unit 3715 (e.g., hard disk), communicationinterface 3720 (e.g., network adapter) for communicating with one ormore other systems, and peripheral devices 3725, such as cache, othermemory, data storage and/or electronic display adapters. The memory3710, storage unit 3715, interface 3720 and peripheral devices 3725 arein communication with the CPU 3705 through a communication bus (solidlines), such as a motherboard. The storage unit 3715 can be a datastorage unit (or data repository) for storing data. The computer system3701 can be operatively coupled to a computer network (“network”) 3730with the aid of the communication interface 3720. The network 3730 canbe the Internet, an internet and/or extranet, or an intranet and/orextranet that is in communication with the Internet. The network 3730 insome cases is a telecommunication and/or data network. The network 3730can include one or more computer servers, which can enable distributedcomputing, such as cloud computing. The network 3730, in some cases withthe aid of the computer system 3701, can implement a peer-to-peernetwork, which may enable devices coupled to the computer system 3701 tobehave as a client or a server.

The CPU 3705 can execute a sequence of machine-readable instructions,which can be embodied in a program or software. The instructions may bestored in a memory location, such as the memory 3710. The instructionscan be directed to the CPU 3705, which can subsequently program orotherwise configure the CPU 3705 to implement methods of the presentdisclosure. Examples of operations performed by the CPU 3705 can includefetch, decode, execute, and writeback.

The CPU 3705 can be part of a circuit, such as an integrated circuit.One or more other components of the system 3701 can be included in thecircuit. In some cases, the circuit is an application specificintegrated circuit (ASIC).

The storage unit 3715 can store files, such as drivers, libraries andsaved programs. The storage unit 3715 can store user data, e.g., userpreferences and user programs. The computer system 3701 in some casescan include one or more additional data storage units that are externalto the computer system 3701, such as located on a remote server that isin communication with the computer system 3701 through an intranet orthe Internet.

The computer system 3701 can communicate with one or more remotecomputer systems through the network 3730. For instance, the computersystem 3701 can communicate with a remote computer system of a user(e.g., a mobile device configured to run one of the recommendationapplications described herein). Examples of remote computer systemsinclude personal computers (e.g., portable PC), slate or tablet PC's(e.g., Apple® iPad, Samsung® Galaxy Tab), telephones, Smart phones(e.g., Apple® iPhone, Android-enabled device, Blackberry®), or personaldigital assistants. The user can access the computer system 3701 via thenetwork 3730.

Methods as described herein can be implemented by way of machine (e.g.,computer processor) executable code stored on an electronic storagelocation of the computer system 3701, such as, for example, on thememory 3710 or electronic storage unit 3715. The machine executable ormachine-readable code can be provided in the form of software. Duringuse, the code can be executed by the processor 3705. In some cases, thecode can be retrieved from the storage unit 3715 and stored on thememory 3710 for ready access by the processor 3705. In some situations,the electronic storage unit 3715 can be precluded, andmachine-executable instructions are stored on memory 3710.

The code can be pre-compiled and configured for use with a machinehaving a processer adapted to execute the code or can be compiled duringruntime. The code can be supplied in a programming language that can beselected to enable the code to execute in a pre-compiled or as-compiledfashion.

Aspects of the systems and methods provided herein, such as the computersystem 3701, can be embodied in programming. Various aspects of thetechnology may be thought of as “products” or “articles of manufacture”typically in the form of machine (or processor) executable code and/orassociated data that is carried on or embodied in a type of machinereadable medium. Machine-executable code can be stored on an electronicstorage unit, such as memory (e.g., read-only memory, random-accessmemory, flash memory) or a hard disk. “Storage” type media can includeany or all of the tangible memory of the computers, processors or thelike, or associated modules thereof, such as various semiconductormemories, tape drives, disk drives and the like, which may providenon-transitory storage at any time for the software programming. All orportions of the software may at times be communicated through theInternet or various other telecommunication networks. Suchcommunications, for example, may enable loading of the software from onecomputer or processor into another, for example, from a managementserver or host computer into the computer platform of an applicationserver. Thus, another type of media that may bear the software elementsincludes optical, electrical and electromagnetic waves, such as usedacross physical interfaces between local devices, through wired andoptical landline networks and over various air-links. The physicalelements that carry such waves, such as wired or wireless links, opticallinks or the like, also may be considered as media bearing the software.As used herein, unless restricted to non-transitory, tangible “storage”media, terms such as computer or machine “readable medium” refer to anymedium that participates in providing instructions to a processor forexecution.

Hence, a machine readable medium, such as computer-executable code, maytake many forms, including but not limited to, a tangible storagemedium, a carrier wave medium or physical transmission medium.Non-volatile storage media include, for example, optical or magneticdisks, such as any of the storage devices in any computer(s) or thelike, such as may be used to implement the databases, etc. shown in thedrawings. Volatile storage media include dynamic memory, such as mainmemory of such a computer platform. Tangible transmission media includecoaxial cables; copper wire and fiber optics, including the wires thatcomprise a bus within a computer system. Carrier-wave transmission mediamay take the form of electric or electromagnetic signals, or acoustic orlight waves such as those generated during radio frequency (RF) andinfrared (IR) data communications. Common forms of computer-readablemedia therefore include for example: a floppy disk, a flexible disk,hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD orDVD-ROM, any other optical medium, punch cards paper tape, any otherphysical storage medium with patterns of holes, a RAM, a ROM, a PROM andEPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wavetransporting data or instructions, cables or links transporting such acarrier wave, or any other medium from which a computer may readprogramming code and/or data. Many of these forms of computer readablemedia may be involved in carrying one or more sequences of one or moreinstructions to a processor for execution.

The computer system 3701 can include or be in communication with anelectronic display 3735 that comprises a user interface (UI) 3740 forproviding, for example, recommendations to a subject (e.g., diet orphysical activity recommendations) that can aid the subject in alteringor maintaining a blood glucose level. Examples of UI's include, withoutlimitation, a graphical user interface (GUI) and web-based userinterface.

Methods and systems of the present disclosure can be implemented by wayof one or more algorithms. An algorithm can be implemented by way ofsoftware upon execution by the central processing unit 3705. Thealgorithm can, for example, any of the machine learning algorithms ormodels described herein.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. It is not intendedthat the invention be limited by the specific examples provided withinthe specification. While the invention has been described with referenceto the aforementioned specification, the descriptions and illustrationsof the embodiments herein are not meant to be construed in a limitingsense. Numerous variations, changes, and substitutions will now occur tothose skilled in the art without departing from the invention.Furthermore, it shall be understood that all aspects of the inventionare not limited to the specific depictions, configurations or relativeproportions set forth herein which depend upon a variety of conditionsand variables. It should be understood that various alternatives to theembodiments of the invention described herein may be employed inpracticing the invention. It is therefore contemplated that theinvention shall also cover any such alternatives, modifications,variations or equivalents. It is intended that the following claimsdefine the scope of the invention and that methods and structures withinthe scope of these claims and their equivalents be covered thereby.

What is claimed is:
 1. A method, comprising: training a neural networkwith time series training data of a first modality and time seriestraining data of a second modality to create a first model thatgenerates time series data of the second modality from time series dataof the first modality; training a second model with the generated timeseries of the second modality, time series training data of a thirdmodality, and time series data of a fourth modality to generate timeseries data of the fourth modality; until a convergence condition isreached, iteratively testing the second model on the time series data ofthe first modality and the time series data of the third modality; andresponsive to reaching the convergence condition, predicting secondmodality data by testing the second model with data of the firstmodality.
 2. The method of claim 1, wherein the time series trainingdata of the second modality has gaps.
 3. The method of claim 1, furthercomprising: acquiring the time series training data of the firstmodality with a first type sensor; and acquiring the time seriestraining data of the second modality with a second type sensor.
 4. Themethod of claim 3, wherein the second type sensor is a glucose meter,and wherein the time series data of the second modality includes glucoselevels over time.
 5. The method of claim 1, wherein training the neuralnetwork to create the first model includes training with N sets of timeseries training data, and wherein training the first model with theestimated time series training data of the first modality and timeseries training data of at least the third modality includes trainingwith M sets of time series data.
 6. The method of claim 5, furthercomprising testing the first model with the N sets of time series dataand the M sets of time series data and updating the first model inresponse to error values of the testing, and wherein the trained firstmodel is the first model with the smallest error.
 7. The method of claim1, wherein reaching a convergence condition includes calculating anerror value not greater than a threshold.
 8. A system, comprising: aninitial model section that includes a first model trained to generatetime series data of a second modality from time series data of a firstmodality with M sets of training data; a training section that includes:a second model derived from the first model and configured to generatetime series data of at least a third modality from at least time seriesdata of a fourth modality with N sets of training data, and a testingsection configured to test the second model with the M and N sets oftraining data, and update the second model in response to test errorvalues; and an inference model that is the second model with the lowesttest error value, configured to infer time series data of the secondmodality from time series data of the first modality.
 9. The system ofclaim 8, wherein the first model, the second model and the inferencemodel comprise neural networks.
 10. The system of claim 8, wherein thetime series data of the first and second modalities are biophysicalsensor data.
 11. The system of claim 10, wherein at least the timeseries data of the first and second modalities are glucose levelscorresponding to glucose meters.
 12. The system of claim 11, wherein thethird and fourth modalities are glucose levels.
 13. The system of claim8, wherein the training section comprises: an inverse model that is aninverse of the first model and configured to generate estimated timeseries data of the first modality from the time series data of the thirdand a fourth modality; an estimator section configured to generatelinear parameters from the estimated time series data of the firstmodality and the time series data of the third modality; sectionconfigured to generate mapped time series data of the first modalityfrom time series data of the third modality using the linear parameters,wherein the second model is trained with the mapped time series data ofthe first modality.
 14. A method, comprising: training a neural networkwith time series training data of a first modality and time seriestraining data of a second modality to create a first model thatgenerates time series data of the second modality from time series dataof the first modality; until a convergence condition is reached: using asecond model to generate estimated time series data of the firstmodality from a mixture of time series data from a third modality and afourth modality, wherein the second model is initiated as an inversemodel of the first model; using the estimated time series data of thefirst modality and time series data of the third modality, training thesecond model to estimate linear fitting parameters; using the estimatedlinear fitting parameters to generate analogous time series data of thefirst modality from the time series data of the third modality; linearlymapping the analogous time series data of the first modality to the timeseries data of the third modality; training a third model using thelinearly mapped analogous time series data from the first modalitymixture of time series data of the third modality and time series dataof the fourth modality to generate a mixture of time series data fromthe third modality and time series data from the fourth modality,wherein the third model is an inverse of the second model; modifying thesecond model to be an inverse model of the third model; and evaluatingwhether the convergence condition has been reached.
 15. The method ofclaim 14, wherein training the third model includes initializing thethird model as the first model.
 16. A method for training a neuralnetwork to calibrate time series data, comprising: receiving calibratedtime series data for a biophysical response and corresponding raw timeseries data for the biophysical response; training, on the calibratedtime series data and the corresponding raw time series data for thebiophysical response, a neural network to generate calibrated timeseries data, which training comprises updating parameters of the neuralnetwork based on a difference between (i) an output of the neuralnetwork for a given raw time series and (ii) a corresponding calibratedtimes series; receiving raw input time series data generated by abiophysical sensor; and generating calibrated time series data byapplying the raw input time series data to the neural network.
 17. Themethod of claim 16, wherein the raw input time series data is generatedby a glucose meter.
 18. The method of claim 16, wherein the neuralnetwork is trained to cancel drift present in the raw input time seriesdata.
 19. The method of claim 18, wherein the raw time series data andraw input time series data are generated by glucose meters.
 20. Themethod of claim 16, wherein training the neural network furthercomprises domain specific feature engineering.
 21. The method of claim16, wherein training the neural network comprises unsupervised training.